View Source Exmld.KinesisStage (exmld v1.0.3)
A GenStage
stage for use in processing data produced by Exmld.KinesisWorker
s.
This module acts as a GenStage producer. Subscribers will receive
Exmld.KinesisStage.Event
s, which each wrap an underlying Exmld.KinesisWorker.Datum
along with information about the producing worker and stage. Downstream processors
should eventually call disposition/2
with the disposition of processing so that
originating workers can checkpoint.
The expected use and workflow is:
Create a stage or set of stages using this module for each distinct processing pipeline. A set of stages could be used by more than one Kinesis stream if the processing being done is the same for all of them.
Create a flow using
Flow.from_stages/2
.Configure the flow using
Exmld.flow/6
.Run the flow, which should run forever.
Configure an erlmld supervision tree with a set of
Exmld.KinesisWorker
s using the stage(s) created in (1).
Link to this section Summary
Functions
Notify stage
of the disposition of processing some items.
Callback implementation for GenStage.handle_call/3
.
Handle subscriber demand.
Callback implementation for GenStage.handle_info/2
.
Callback implementation for GenStage.init/1
.
Notify stage
of a new Kinesis record available for processing.
Link to this section Functions
Notify stage
of the disposition of processing some items.
An attempt has been made to process some data extracted from a Kinesis record by a
downstream processor. stage
will look up the originating producer and record the
disposition of processing in the next batch of data to be returned to that producer.
Callback implementation for GenStage.handle_call/3
.
Handle subscriber demand.
Return up to incoming_demand + pending_demand
events, fetching (from state) as needed,
and storing in state any excess. If not enough events are available, record unsatisfied
demand in state, and then return those events when answering a subsequent call. See the
QueueBroadcaster
example in GenStage
for an explanation of this demand queueing
behavior.
Callback implementation for GenStage.handle_info/2
.
Callback implementation for GenStage.init/1
.
@spec notify( GenStage.stage(), Exmld.KinesisWorker.Datum, :infinity | non_neg_integer() ) :: {:disposition, [Exmld.KinesisWorker.Disposition.t()]}
Notify stage
of a new Kinesis record available for processing.
A new event is available for processing by stage
. The caller will be monitored and
associated with the new event, and will be blocked until after the event has been used
to satisfy some downstream demand. The return value will be the disposition
(success/failure) of zero or more records which were previously processed.