exmld v0.1.4 Exmld.KinesisStage View Source
A GenStage
stage for use in processing data produced by Exmld.KinesisWorker
s.
This module acts as a GenStage producer. Subscribers will receive
Exmld.KinesisStage.Event
s, which each wrap an underlying Exmld.KinesisWorker.Datum
along with information about the producing worker and stage. Downstream processors
should eventually call disposition/2
with the disposition of processing so that
originating workers can checkpoint.
The expected use and workflow is:
Create a stage or set of stages using this module for each distinct processing pipeline. A set of stages could be used by more than one Kinesis stream if the processing being done is the same for all of them.
Create a flow using
Flow.from_stages/2
.Configure the flow using
Exmld.flow/6
.Run the flow, which should run forever.
Configure an erlmld supervision tree with a set of
Exmld.KinesisWorker
s using the stage(s) created in (1).
Link to this section Summary
Functions
Notify stage
of the disposition of processing some items
Handle subscriber demand
Invoked when the server is started
Notify stage
of a new Kinesis record available for processing
Link to this section Functions
Notify stage
of the disposition of processing some items.
An attempt has been made to process some data extracted from a Kinesis record by a
downstream processor. stage
will look up the originating producer and record the
disposition of processing in the next batch of data to be returned to that producer.
Handle subscriber demand.
Return up to incoming_demand + pending_demand
events, fetching (from state) as needed,
and storing in state any excess. If not enough events are available, record unsatisfied
demand in state, and then return those events when answering a subsequent call. See the
QueueBroadcaster
example in GenStage
for an explanation of this demand queueing
behavior.
Invoked when the server is started.
start_link/3
(or start/3
) will block until this callback returns.
args
is the argument term (second argument) passed to start_link/3
(or start/3
).
In case of successful start, this callback must return a tuple where the first element is the stage type, which is one of:
:producer
:consumer
:producer_consumer
(if the stage is acting as both)
For example:
def init(args) do
{:producer, some_state}
end
The returned tuple may also contain 3 or 4 elements. The third
element may be the :hibernate
atom or a set of options defined
below.
Returning :ignore
will cause start_link/3
to return :ignore
and the process will exit normally without entering the loop or
calling terminate/2
.
Returning {:stop, reason}
will cause start_link/3
to return
{:error, reason}
and the process to exit with reason reason
without entering the loop or calling terminate/2
.
Options
This callback may return options. Some options are specific to the chosen stage type while others are shared across all types.
:producer
options
:demand
- when:forward
, the demand is always forwarded to thec:handle_demand/2
callback. When:accumulate
, demand is accumulated until its mode is set to:forward
viademand/2
. This is useful as a synchronization mechanism, where the demand is accumulated until all consumers are subscribed. Defaults to:forward
.
:producer
and :producer_consumer
options
:buffer_size
- the size of the buffer to store events without demand. Can be:infinity
to signal no limit on the buffer size. Check the “Buffer events” section of the module documentation. Defaults to10_000
for:producer
,:infinity
for:producer_consumer
.:buffer_keep
- returns whether the:first
or:last
entries should be kept on the buffer in case the buffer size is exceeded. Defaults to:last
.:dispatcher
- the dispatcher responsible for handling demands. Defaults toGenStage.DemandDispatch
. May be either an atom representing a dispatcher module or a two-element tuple with the dispatcher module and the dispatcher options.
:consumer
and :producer_consumer
options
:subscribe_to
- a list of producers to subscribe to. Each element represents either the producer module or a tuple with the producer module and the subscription options (as defined insync_subscribe/2
).
Callback implementation for GenStage.init/1
.
notify(GenStage.stage(), Exmld.KinesisWorker.Datum, :infinity | non_neg_integer()) :: {:disposition, [Exmld.KinesisWorker.Disposition.t()]}
Notify stage
of a new Kinesis record available for processing.
A new event is available for processing by stage
. The caller will be monitored and
associated with the new event, and will be blocked until after the event has been used
to satisfy some downstream demand. The return value will be the disposition
(success/failure) of zero or more records which were previously processed.