Writing Events

This guide covers specifics about the Spear.append/4 function and general information about the event-writing functionality.

Enumeration Characteristics

event_stream is an Enumerable.t/0 which will be lazily written to the EventStoreDB as elements of the stream are computed and serialized on to the wire.

This means a few things:

First, you can efficiently emit events from a stream over a large source such as a CSV file with many lines:

File.stream!("large.csv", read_ahead: 100_000)
|> MyCsvParser.parse_stream()
|> Stream.map(&MyCsvParser.turn_csv_line_into_spear_event/1)
|> Spear.append(conn, "ChargesFromCsvs", timeout: :infinity)
# => :ok

The stream is only fully run after the last bytes have been written to the gRPC network request: the stream is never computed entirely in memory.

Second, you may (but are not encouraged to) write events via an infinite stream. A trivial counter mechanism could be implemented like so

iex> Stream.iterate(0, &(&1 + 1))
...> |> Stream.map(fn n -> Spear.Event.new("incremented", n) end)
...> |> Spear.append(conn, "InfiniteCounter", timeout: :infinity, expect: :empty)
{:error,
 %Spear.Grpc.Response{
   data: "",
   message: "Maximum Append Size of 1048576 Exceeded.",
   status: :invalid_argument,
   status_code: 3
 }}

Note that while EventStoreDB streams can in theory store infinitely long streams, they are not practically able to do so. EventStoreDB limits the size of a single write to 1_048_576 cumulative bytes. This budget can be spent on one very large event or, as shown above, many tiny events in a single call to Spear.append/4. Attempting to write more than the budget will fail the request with the above signature and no events in the request will be written to the EventStoreDB. This value is configurable in the EventStoreDB configuration.

Blocking

While Spear.append/4 blocks the caller for the duration of the request, it does not fully block the connection. The connection will write chunks of data over the wire as allowed by HTTP2 window sizes.

HTTP2 includes a back-pressure mechanism for clients sending large amounts of data to the server faster than the server can handle. Servers negotiate a maximum number of bytes which the client is allowed to send called a window. When the window has been exhausted by streaming data to the server, the client must wait until the server replenishes the window. During the downtime between exhausting a window and waiting for the server to replenish, Spear suspends the exhausted request stream and handles incoming messages from the server as normal. Since HTTP2 window sizes are relatively small (usually somewhere around the range of 10 and 100 KB), Spear takes conceptual breaks somewhat often during large requests. This allows Spear to efficiently multiplex large writes with large reads and subscriptions.

Batching

When appending multiple events, Spear will fit as many messages as possible into the same HTTP2 DATA frame. This is valid according to the gRPC specification and has the potential to improve performance.