Module kflow_gen_buffer

This module implements a stateful buffer-like stream processing node.

Copyright © 2019 Klarna Bank AB (publ)

This module defines the kflow_gen_buffer behaviour.
Required callback functions: in/4, out/2.
Optional callback functions: init/1, terminate/2.

Description

This module implements a stateful buffer-like stream processing node.

One has to create a callback module with kflow_gen_buffer behavior.

init and terminate callbacks are similar to those in kflow_gen_map or kflow_gen_filter behaviors.

Consuming upstream messages

in callback is invoked for each incoming message from the upstream. It takes 4 arguments:

  1. Offset of a message
  2. Message itself
  3. State of the callback module. This state is created in init callback and can be mutated in the callbacks.
  4. Last argument is initial configuration of the buffer (constant)

Return value should be a tuple {Flush, NextState} where Flush can be atoms keep, flush or reject.

keep means that the buffer should keep collecting upstream messages without producing anything downstream.

flush means that the buffer is ready to produce a message downstream.

reject means the last upstream message was incompatible with the data that had been buffered so far. (E.g. schema of the data was different). In this case previous state is flushed and the last message is replayed from blank state.

Producing messages downstream

out callback is used to produce a message downstream. It is invoked after in callback returns flush or reject, or when flush is implicitly requested by low-level control logic. It takes two arguments: first one is current state of the callback module and the second one is initial configuration.

It should output a tuple {DownstreamMessage, NextState}

Example

   -module(my_map).
  
   -behavior(kflow_gen_map).
  
   -export([init/1, map/3, terminate/1]).
  
   init(_Config) ->
     [].
  
   in(Offset, Message, State, Config) ->
     N = maps:get(buffer_size, Config),
     Flush = if length(State) >= N ->
                  flush;
                true ->
                  keep
             end,
     {Flush, [Message|State]}.
  
   out(State, _Config) ->
     Output = lists:reverse(State),
     NewState = [],
     {Output, NewState}.
  
   terminate(_State) ->
     ok.

Generated by EDoc