Spear.stream-exclamation-mark

You're seeing just the function stream-exclamation-mark, go back to Spear module for more information.
Link to this function

stream!(connection, stream_name, opts \\ [])

View Source (since 0.1.0)

Specs

stream!(
  connection :: Spear.Connection.t(),
  stream_name :: String.t() | :all,
  opts :: Keyword.t()
) :: event_stream :: Enumerable.t()

Collects an EventStoreDB stream into an enumerable

This function may raise in cases where the gRPC requests fail to read events from the EventStoreDB (in cases of timeout or unavailability).

This function does not raise if a stream does not exist (is empty), instead returning an empty enumerable [].

connection may be any valid GenServer name (including PIDs) for a process running the Spear.Connection GenServer.

stream_name can be any stream, existing or not, including projected streams such as category streams or event-type streams. The :all atom may be passed as stream_name to read all events in the EventStoreDB.

Options

  • :from - (default: :start) the EventStoreDB stream revision from which to read. Valid values include :start, :end, any non-negative integer representing the event number revision in the stream and events. Event numbers are inclusive (e.g. reading from 0 will first return the event with revision 0 in the stream, if one exists). :start and :end are treated as inclusive (e.g. :start will return the first event in the stream). Events (either Spear.Event or ReadResp records) can also be supplied and will be treated as inclusive.
  • :direction - (default: :forwards) the direction in which to read the EventStoreDB stream. Valid values include :forwards and :backwards. Reading the EventStoreDB stream forwards will return events in the order in which they were written to the EventStoreDB; reading backwards will return events in the opposite order.
  • :resolve_links? - (default: true) whether or not to request that link references be resolved. See the moduledocs for more information about link resolution.
  • :chunk_size - (default: 128) the number of events to read from the EventStoreDB at a time. Any positive integer is valid. See the enumeration characteristics section below for more information about how :chunk_size works and how to tune it.
  • :timeout - (default: 5_000 - 5s) the time allowed for the read of a single chunk of events in the EventStoreDB stream. This time is not cumulative: an EventStoreDB stream 100 events long which takes 5s to read each chunk may be read in chunks of 20 events culumaltively in 25s. A timeout of 5_001ms would not raise a timeout error in that scenario (assuming the chunk read consistently takes <= 5_000 ms).
  • :raw?: - (default: false) controls whether or not the enumerable event_stream is decoded to Spear.Event structs from their raw Spear.Records.Streams.read_resp/0 output. Setting raw?: true prevents this transformation and leaves each event as a ReadResp record. See Spear.Event.from_read_response/2 for more information.
  • :credentials - (default: nil) a two-tuple {username, password} to use as credentials for the request. This option overrides any credentials set in the connection configuration, if present. See the Security guide for more details.

Enumeration Characteristics

The event_stream Enumerable.t/0 returned by this function initially contains a buffer of bytes from the first read of the stream stream_name. This buffer potentially contains up to :chunk_size messages when run. The enumerable is written as a formula which abstracts away the chunking nature of the gRPC requests, however, so even though the EventStoreDB stream is read in chunks (per the :chunk_size option), the entire EventStoreDB stream can be read by running the enumeration (e.g. with Enum.to_list/1). Note that the stream will make a gRPC request to read more events whenever the buffer runs dry with up to :chunk_size messages filling the buffer on each request.

:chunk_size is difficult to tune as it causes a tradeoff between (gRPC) request duration and number of messages added to the buffer. A higher :chunk_size may hydrate the buffer with more events and reduce the number of gRPC requests needed to read an entire stream, but it also increases the number of messages that will be sent over the network per request which could decrease reliability. Generally speaking, a lower :chunk_size is appropriate for streams in which the events are large and a higher :chunk_size is appropriate for streams with many small events. Manual tuning and trial-and-error can be used to find a performant :chunk_size setting for any individual environment.

Examples

iex> Spear.stream!(MyConnection, "es_supported_clients", chunk_size: 1) |> Enum.take(1)
[
  %Spear.Event{
    body: %{"languages" => ["typescript", "javascript"], "runtime" => "NodeJS"},
    id: "1fc908c1-af32-4d06-a9bd-3bf86a833fdf",
    metadata: %{..},
    type: "grpc-client"
  }
]
# say we have 5 events in the "es_supported_clients" stream
iex> Spear.stream!(MyConnection, "es_supported_clients", chunk_size: 3) |> Enum.count()
5