View Source Exmld.KinesisStage (exmld v1.0.4)

A GenStage stage for use in processing data produced by Exmld.KinesisWorkers.

This module acts as a GenStage producer. Subscribers will receive Exmld.KinesisStage.Events, which each wrap an underlying Exmld.KinesisWorker.Datum along with information about the producing worker and stage. Downstream processors should eventually call disposition/2 with the disposition of processing so that originating workers can checkpoint.

The expected use and workflow is:

  1. Create a stage or set of stages using this module for each distinct processing pipeline. A set of stages could be used by more than one Kinesis stream if the processing being done is the same for all of them.

  2. Create a flow using Flow.from_stages/2.

  3. Configure the flow using Exmld.flow/6.

  4. Run the flow, which should run forever.

  5. Configure an erlmld supervision tree with a set of Exmld.KinesisWorkers using the stage(s) created in (1).

Link to this section Summary

Functions

Notify stage of the disposition of processing some items.

Handle subscriber demand.

Callback implementation for GenStage.handle_info/2.

Callback implementation for GenStage.init/1.

Notify stage of a new Kinesis record available for processing.

Link to this section Functions

Link to this function

disposition(stage, disposition, timeout \\ :infinity)

View Source

Notify stage of the disposition of processing some items.

An attempt has been made to process some data extracted from a Kinesis record by a downstream processor. stage will look up the originating producer and record the disposition of processing in the next batch of data to be returned to that producer.

Link to this function

handle_call(arg, from, state)

View Source

Callback implementation for GenStage.handle_call/3.

Link to this function

handle_demand(incoming_demand, state)

View Source

Handle subscriber demand.

Return up to incoming_demand + pending_demand events, fetching (from state) as needed, and storing in state any excess. If not enough events are available, record unsatisfied demand in state, and then return those events when answering a subsequent call. See the QueueBroadcaster example in GenStage for an explanation of this demand queueing behavior.

Callback implementation for GenStage.handle_info/2.

Callback implementation for GenStage.init/1.

Link to this function

notify(stage, datum, timeout \\ :infinity)

View Source
@spec notify(
  GenStage.stage(),
  Exmld.KinesisWorker.Datum,
  :infinity | non_neg_integer()
) :: {:disposition, [Exmld.KinesisWorker.Disposition.t()]}

Notify stage of a new Kinesis record available for processing.

A new event is available for processing by stage. The caller will be monitored and associated with the new event, and will be blocked until after the event has been used to satisfy some downstream demand. The return value will be the disposition (success/failure) of zero or more records which were previously processed.