RecoverableStreamEx v1.0.0 RecoverableStream View Source

By extracting evaluation of the source stream into a separate process RecoverableStream provides a way to isolate upstream errors and recover from them.

This module contains public API.

Link to this section Summary

Functions

Evaluates passed stream_fun/0 inside a new Task then runs produced stream, forwarding data back to the caller.

Link to this section Types

Link to this type

inner_reduce_fun()

View Source
inner_reduce_fun() :: (stream_arg_t() -> none())
Link to this type

last_value_t()

View Source
last_value_t() :: nil | any()
Link to this type

run_option()

View Source
run_option() ::
  {:retry_attempts, non_neg_integer()}
  | {:wrapper_fun, wrapper_fun()}
  | {:task_supervisor, atom() | pid()}
Link to this type

stream_arg_t()

View Source
stream_arg_t() :: any()
Link to this type

wrapper_fun()

View Source
wrapper_fun() :: (inner_reduce_fun() -> none())

Link to this section Functions

Link to this function

run(new_stream_fun, options \\ [])

View Source

Evaluates passed stream_fun/0 inside a new Task then runs produced stream, forwarding data back to the caller.

Returns a new Stream that gathers data forwarded by the Task. Data is forwarded element by element. Batching is to be implemented explicitly. For example Postgrex.stream/3 sends data in chunks by default.

Stream function

stream_fun/0 must be a function that accepts one or two arguments.

  • The first argument is either nil or the last value received from a stream before recovery.
  • The second argument is an arbitrary term passed from wrapper_fun/0

The function should return a Stream (although, any Enumerable could work).

Example

iex> gen_stream_f = fn
...>   nil -> Stream.iterate(1, fn x when x < 2 -> x + 1 end)
...>     x -> Stream.iterate(x + 1, &(&1+1))
...> end
iex> RecoverableStream.run(gen_stream_f)
...> |> Stream.take(4)
...> |> Enum.into([])
[1, 2, 3, 4]

Options

  • :retry_attempts (defaults to 1) the total number of times error recovery is performed before an error is propagated.

    Retries counter is not reset upon a successful recovery!

  • :task_supervisor either pid or a name of Task.Supervisor to supervise a stream-reducer Task. (defaults to RecoverableStream.TaskPool)

    See RecoverableStream.TasksPool.child_spec/1 for details.

  • :wrapper_fun is a funciton that wraps a stream reducer running inside a Task (defaults to fun f -> f.(%{}) end).

    Useful when the stream_fun/0 must be run within a certain context. E.g. Postgrex.stream/3 only works inside Postgrex.transaction/3.

    See Readme for a more elaborate example.