RecoverableStreamEx v1.0.0 RecoverableStream View Source
By extracting evaluation of the source stream into a separate process
RecoverableStream
provides a way to isolate upstream errors
and recover from them.
This module contains public API.
Link to this section Summary
Functions
Evaluates passed stream_fun/0
inside a new Task
then runs
produced stream, forwarding data back to the caller.
Link to this section Types
run_option()
View Sourcerun_option() :: {:retry_attempts, non_neg_integer()} | {:wrapper_fun, wrapper_fun()} | {:task_supervisor, atom() | pid()}
stream_fun()
View Sourcestream_fun() :: (last_value_t() -> Enumerable.t()) | (last_value_t(), stream_arg_t() -> Enumerable.t())
Link to this section Functions
run(new_stream_fun, options \\ [])
View Sourcerun(stream_fun(), [run_option()]) :: Enumerable.t()
Evaluates passed stream_fun/0
inside a new Task
then runs
produced stream, forwarding data back to the caller.
Returns a new Stream
that gathers data forwarded by the Task
.
Data is forwarded element by element. Batching is to be implemented
explicitly. For example Postgrex.stream/3
sends data in chunks
by default.
Stream function
stream_fun/0
must be a function that accepts one or two arguments.
- The first argument is either
nil
or the last value received from a stream before recovery. - The second argument is an arbitrary term passed from
wrapper_fun/0
The function should return a Stream
(although, any Enumerable
could work).
Example
iex> gen_stream_f = fn
...> nil -> Stream.iterate(1, fn x when x < 2 -> x + 1 end)
...> x -> Stream.iterate(x + 1, &(&1+1))
...> end
iex> RecoverableStream.run(gen_stream_f)
...> |> Stream.take(4)
...> |> Enum.into([])
[1, 2, 3, 4]
Options
:retry_attempts
(defaults to1
) the total number of times error recovery is performed before an error is propagated.Retries counter is not reset upon a successful recovery!
:task_supervisor
either pid or a name ofTask.Supervisor
to supervise a stream-reducerTask
. (defaults toRecoverableStream.TaskPool
)See
RecoverableStream.TasksPool.child_spec/1
for details.:wrapper_fun
is a funciton that wraps a stream reducer running inside aTask
(defaults tofun f -> f.(%{}) end
).Useful when the
stream_fun/0
must be run within a certain context. E.g.Postgrex.stream/3
only works insidePostgrex.transaction/3
.See Readme for a more elaborate example.