Sheetshow.Log (Sheetshow v0.1.0)

Copy Markdown View Source

An append-only tab, and the first of the two ways to keep data on a sheet.

A log's rows are events, each carrying an id its writer made. Nothing is ever overwritten: an update is the same id appended again, a delete is the same id appended with the deleted flag, and the state of the log is the fold over the rows: the last one per id wins, and tombstoned ids drop out.

iex> alias Sheetshow.{Log, Workbook}
iex> log = Log.new("expenses", item: :string, cost: :decimal)
iex> rent = Log.Event.new(%{item: "Rent", cost: "1000.00"}, id: "a")
iex> food = Log.Event.new(%{item: "Food", cost: "400.00"}, id: "b")
iex> {:ok, workbook} = Sheetshow.run(Log.create(log) ++ Log.plan!([rent, food], log), Workbook.memory())
iex> later = [Log.Event.put(rent, %{cost: "1100.00"}), Log.Event.delete(food)]
iex> {:ok, workbook} = Sheetshow.run(Log.plan!(later, log), workbook)
iex> Log.read!(log, workbook) |> Log.fold() |> Enum.map(& &1.record)
[%{item: "Rent", cost: "1100.00"}]

That one rule does three jobs. It makes updates and deletes out of the only write Google resolves for you, since appendCells lands below the last row with data at the moment it is applied, so two processes appending to one tab cannot overwrite each other, and no compare-and-swap is needed. And it makes a write safe to retry: a batch that landed but answered with a transport error can be sent again, because the duplicate ids fold away. Retry a failed batch before you write anything newer, though, because a stale retry that lands late is a stale last row.

That is Google's promise, not a file's. A file is rewritten whole, so two writers appending to one can lose each other's rows unless the store can refuse a clobbering write, in which case the loser gets %Sheetshow.Error{reason: :conflict} with nothing written, and running the same plan again is safe, because the ids in it fold away against themselves. What Sheetshow Can Promise has the whole of it.

The tab is laid out for a person to read. Row 0 is the header (id, deleted, then the schema's columns in order), and deleted is left blank on a live row, so a reader sees a flag only where something was taken out. Columns are found by header name when reading, but appends are positional, so the schema's order is the tab's column order: add columns at the end and re-put header/2, and leave the ones already there where they are. Anything further right is a person's own business and is not touched.

Reading is lenient. A cell someone typed that will not cast leaves that field nil and an entry in the event's errors, so a typo costs you a field rather than the read; decode/3 with strict: true refuses such a read instead.

Summary

Functions

The tab's columns, left to right.

The plan that makes the tab and writes its header. Fails at the backend if the tab is already there, which is what you want from something called create.

Events from the tab's values: rows of cell values, as Sheetshow.read_cells/2 or Sheetshow.to_rows/1 gives them.

Same as decode/3, raising on failure.

The log as it stands: the latest event for each id, in the order the ids first appeared, with the deleted ones gone.

The header as cells, bold unless you say otherwise. Write these again after adding a column to the end of the schema.

A log on a tab, with the columns its rows have. Raises ArgumentError on a schema Sheetshow.Schema.validate/1 refuses.

The plan that appends events, as one Sheetshow.Op.AppendRows, so a batch of them is atomic and lands below whatever else arrived in the meantime.

Same as plan/2, raising on failure.

Every event on the tab, in the order it was written.

Same as read/3, raising on failure.

Cells for a second tab that shows the log as it stands, for a person to look at: one live row per id, newest content, tombstones gone.

Types

t()

@type t() :: %Sheetshow.Log{schema: Sheetshow.Schema.t(), sheet: String.t()}

Functions

columns(log)

@spec columns(t()) :: [String.t()]

The tab's columns, left to right.

iex> Sheetshow.Log.new("expenses", item: :string) |> Sheetshow.Log.columns()
["id", "deleted", "item"]

create(log)

@spec create(t()) :: Sheetshow.Op.plan()

The plan that makes the tab and writes its header. Fails at the backend if the tab is already there, which is what you want from something called create.

iex> Sheetshow.Log.new("expenses", item: :string) |> Sheetshow.Log.create() |> Enum.map(&Sheetshow.Op.sheet/1)
["expenses", "expenses"]

decode(rows, log, opts \\ [])

@spec decode([[term()]], t(), keyword()) ::
  {:ok, [Sheetshow.Log.Event.t()]} | {:error, Sheetshow.Error.t()}

Events from the tab's values: rows of cell values, as Sheetshow.read_cells/2 or Sheetshow.to_rows/1 gives them.

Options:

  • :header, the header row, when rows holds only data. Without it the first row of rows is the header.
  • :row, the sheet row the first given row sits on, so events know where they are. Defaults to 0 with a header in rows, 1 without.
  • :strict, to refuse the read when any cell would not cast, rather than flagging the field. Off by default.

Columns are matched by header name, ignoring case and surrounding space, so a column that has been renamed fails loudly rather than reading as the one next to it. Blank rows are skipped; a row with content and no id comes back as an event with errors.id, because dropping it would hide data someone typed.

iex> log = Sheetshow.Log.new("expenses", item: :string)
iex> rows = [["id", "deleted", "item"], ["a", nil, "Rent"], ["a", "TRUE", "Rent"]]
iex> {:ok, events} = Sheetshow.Log.decode(rows, log)
iex> Enum.map(events, &{&1.id, &1.row, &1.deleted})
[{"a", 1, false}, {"a", 2, true}]

iex> log = Sheetshow.Log.new("expenses", cost: :integer)
iex> {:error, %Sheetshow.Error{reason: :missing_column}} =
...>   Sheetshow.Log.decode([["id", "deleted", "price"]], log)

decode!(rows, log, opts \\ [])

@spec decode!([[term()]], t(), keyword()) :: [Sheetshow.Log.Event.t()]

Same as decode/3, raising on failure.

iex> log = Sheetshow.Log.new("expenses", item: :string)
iex> Sheetshow.Log.decode!([["id", "deleted", "item"]], log)
[]

fold(events)

The log as it stands: the latest event for each id, in the order the ids first appeared, with the deleted ones gone.

iex> alias Sheetshow.Log
iex> log = [Log.Event.new(%{}, id: "a"), Log.Event.new(%{}, id: "b"), Log.Event.delete("a")]
iex> Log.fold(log) |> Enum.map(& &1.id)
["b"]

One pass over a map, so a hundred thousand rows fold in one go, and the result folds again: fold(fold(old) ++ new) is fold(old ++ new), because the folded list keeps both the order the ids first appeared in and the latest content of each. So a process reading a log in batches can merge each read into what it holds instead of re-folding the lot. The one exception is an id that was deleted and then written again: it lands where it came back rather than where it started.

A row whose id would not read takes no part in any of this, and cannot be updated or deleted by id, but it is kept, under its own row, so that data a person typed does not vanish from the fold.

header(log, style \\ %{bold: true})

@spec header(t(), Sheetshow.Style.t()) :: [Sheetshow.Cell.t()]

The header as cells, bold unless you say otherwise. Write these again after adding a column to the end of the schema.

iex> Sheetshow.Log.new("expenses", item: :string)
...> |> Sheetshow.Log.header()
...> |> Sheetshow.to_rows()
[["id", "deleted", "item"]]

new(sheet, schema)

@spec new(String.t(), Sheetshow.Schema.t()) :: t()

A log on a tab, with the columns its rows have. Raises ArgumentError on a schema Sheetshow.Schema.validate/1 refuses.

iex> Sheetshow.Log.new("expenses", item: :string, cost: :decimal).sheet
"expenses"

plan(events, log)

@spec plan([Sheetshow.Log.Event.t()], t()) ::
  {:ok, Sheetshow.Op.plan()} | {:error, Sheetshow.Error.t()}

The plan that appends events, as one Sheetshow.Op.AppendRows, so a batch of them is atomic and lands below whatever else arrived in the meantime.

Strict, as writing always is: a record whose values do not fit the schema, or which names a column the schema does not have, is an error here rather than a surprise in the sheet. An empty list is an empty plan.

iex> log = Sheetshow.Log.new("expenses", item: :string)
iex> {:ok, [op]} = Sheetshow.Log.plan([Sheetshow.Log.Event.new(%{item: "Rent"})], log)
iex> Sheetshow.Op.AppendRows.height(op)
1

iex> log = Sheetshow.Log.new("expenses", cost: :integer)
iex> {:error, %Sheetshow.Error{reason: :invalid_record}} =
...>   Sheetshow.Log.plan([Sheetshow.Log.Event.new(%{cost: "free"})], log)

plan!(events, log)

@spec plan!([Sheetshow.Log.Event.t()], t()) :: Sheetshow.Op.plan()

Same as plan/2, raising on failure.

iex> log = Sheetshow.Log.new("expenses", item: :string)
iex> Sheetshow.Log.plan!([], log)
[]

read(log, workbook, opts \\ [])

@spec read(t(), Sheetshow.Workbook.t(), keyword()) ::
  {:ok, [Sheetshow.Log.Event.t()]} | {:error, Sheetshow.Error.t()}

Every event on the tab, in the order it was written.

This is Sheetshow.read_rows/2 over the whole tab and then decode/3, and it takes decode/3's :strict. The values endpoint rather than the cell one because a stored row wants what its formulas worked out to, and because the schema already says which numbers are dates. See Sheetshow.read_rows/2 for what that costs you.

Reading the whole tab is what a fold needs; Google trims the answer to the rows that have something in them, so an empty log is one small request. A file is read whole whatever it holds.

{:ok, workbook} = Sheetshow.connect(workbook)
{:ok, events} = Sheetshow.Log.read(log, workbook)
Sheetshow.Log.fold(events)

Reading only what is new

after: event answers with the events appended since that one, which is what a log too big to read every time needs. The cursor is an event you were given by a read, since its row is what makes it one, and the usual cursor is the last event you hold.

{:ok, new} = Sheetshow.Log.read(log, workbook, after: List.last(events))
state = Sheetshow.Log.fold(Sheetshow.Log.fold(events) ++ new)

The read covers the cursor's own row as well as the rows below it, and the cursor's row must still hold the very same event. If it does not, because someone inserted or removed a row above it by hand, the rows you hold no longer line up with the sheet, and you get {:error, %Error{reason: :moved}} rather than a list that quietly means something else. Read the log whole when that happens.

Two things this cannot see. A row appended and then deleted by hand below the cursor is simply gone, and nothing says so. And two neighbouring rows that are identical in every respect can stand in for each other, which is harmless, because folding either of them gives the same answer.

At Google it is one request: the header row and the rows from the cursor down, asked for together, so the columns are still found by name rather than assumed. Against a file it is the file: a cursor saves the decoding, not the fetching, because a workbook has to be read whole before any of it can be.

read!(log, workbook, opts \\ [])

@spec read!(t(), Sheetshow.Workbook.t(), keyword()) :: [Sheetshow.Log.Event.t()]

Same as read/3, raising on failure.

view(log, sheet)

@spec view(t(), String.t()) :: [Sheetshow.Cell.t()]

Cells for a second tab that shows the log as it stands, for a person to look at: one live row per id, newest content, tombstones gone.

It is a single formula, so Sheets keeps it current as the log grows, and nothing has to be rewritten when rows are appended. The deleted column is left out, since every row on this tab is one that was not.

log |> Sheetshow.Log.view("expenses now") |> Sheetshow.plan!(existing_sheets: [])

The formula is what fold/1 does, said in Sheets: drop the blank rows, put the log newest-first, keep one row per id, drop the deleted ones, and order what is left by id.

The same rows, in id order. fold/1 keeps the order the ids first appeared in; this tab can only sort by what is on it, which is the id. Those agree when your ids increase in the order you make them, which a Sheetshow.ULID does down to the microsecond, so a batch of events made in one go reads back here in the order it was made. Ids of your own need to sort the same way if the order on this tab has to mean something.

Said in Sheets, and only there. It turns on SORTN, which Excel has no counterpart for, so a view tab written into an .xlsx shows #NAME? when somebody opens it. Nothing is harmed, since a formula that does not resolve is a cell that does not resolve, but the tab is pointless there, and fold/1 is what a file-backed log uses instead.

iex> Sheetshow.Log.new("log", item: :string)
...> |> Sheetshow.Log.view("now")
...> |> Sheetshow.to_rows()
...> |> hd()
["id", "item"]