batch_please v0.2.0 BatchPlease behaviour
BatchPlease is a tool for collecting batches of items, and doing something with each batch when it reaches a certain size or age.
It is built on top of GenServer, implemented as a behaviour,
and invoked through use BatchPlease
.
It is useful to build specialized batch collectors on top of BatchPlease,
in order to abstract more details from the end user. Examples of this
approach include BatchPlease.MemoryBatcher
(which stores items in memory
when batching) and BatchPlease.FileBatcher
(which encodes items to
string format and accumulates them in an on-disk file until ready for
processing).
Simple/trivial usage example:
defmodule Summer do
use BatchPlease, max_batch_size: 3
def batch_init(_opts) do
{:ok, %{sum: 0}}
end
def batch_add_item(batch, item) do
{:ok, %{batch | sum: batch.sum + item}}
end
def batch_process(batch) do
IO.puts("This batch added up to #{batch.sum}")
:ok
end
end
{:ok, pid} = GenServer.start_link(Summer, [])
BatchPlease.add_item(pid, 1)
BatchPlease.add_item(pid, 2)
BatchPlease.add_item(pid, 3) # prints "This batch added up to 6"
BatchPlease.add_item(pid, 4)
BatchPlease.add_item(pid, 5)
BatchPlease.add_item(pid, 6) # prints "This batch added up to 15"
Link to this section Summary
Types
A map representing the state of the current batch. Contents of this map are implementation-specific
Represents the return value of functions which generate a new batch state
(batch_init/1
and batch_add_item/2
)
A GenServer performing as a batch server
Any item that can be added to a batch
Return value of functions which may fail, but do not return a new batch
state (batch_process/1
and batch_terminate/1
)
Configuration parameters for a batch server
A map representing the internal state of a batch server. This map
contains a batch
key, representing the state of the current batch
Functions
Adds an item to a batch.
Returns :ok
on success, or {:error, message}
otherwise
Forces the processing and flushing of a batch.
Returns :ok
on success, or {:error, message}
otherwise
Callbacks
Adds an item to the batch represented in batch
Creates a new batch state, given the configuration options in opts
.
This function is called not just once, but every time a new batch
is created (i.e., at startup and after every flush).
Defaults to creating an empty map, returning {:ok, %{}}
Performs some post-processing on the batch, after batch_process/1
has completed successfully. Does not return an updated batch, because
this operation is immediately followed by batch_init/1
to create a new
batch
Performs some pre-processing on the batch, before it is passed to
batch_process/1
. Returns the updated batch state or an error message
Processes the batch, whatever that entails
Cleans up batch state before the batcher process is terminated. Defaults to no-op. This is not guaranteed to be called at termination time — for more information, see: https://hexdocs.pm/elixir/GenServer.html#c:terminate/2
Given the current module state, returns whether the current batch
should be processed now. Precedes the handling of max_batch_size
(but does not replace it)
Link to this section Types
A map representing the state of the current batch. Contents of this map are implementation-specific.
Represents the return value of functions which generate a new batch state
(batch_init/1
and batch_add_item/2
).
A GenServer performing as a batch server.
Any item that can be added to a batch.
Return value of functions which may fail, but do not return a new batch
state (batch_process/1
and batch_terminate/1
).
option :: {:eager_flush, boolean} | {:max_batch_size, non_neg_integer | nil} | {:max_time_since_last_flush, non_neg_integer | nil} | {:max_time_since_first_item, non_neg_integer | nil}
Configuration parameters for a batch server.
state() :: %{opts: opts, module: atom, batch: batch, last_item: item | nil, config: state_config, overrides: state_overrides, counts: state_counts, times: state_times}
A map representing the internal state of a batch server. This map
contains a batch
key, representing the state of the current batch.
state_config() :: %{lazy_flush: boolean | nil, max_batch_size: non_neg_integer | nil, max_time_since_last_flush: non_neg_integer | nil, max_time_since_first_item: non_neg_integer | nil}
state_counts() :: %{batch_items: non_neg_integer, total_items: non_neg_integer, flushes: non_neg_integer}
state_overrides() :: %{batch_init: (opts -> batch_return) | nil, batch_add_item: (batch, item -> batch_return) | nil, batch_pre_process: (batch -> batch_return) | nil, batch_process: (batch -> ok_or_error) | nil, batch_post_process: (batch -> ok_or_error) | nil, batch_terminate: (batch -> ok_or_error) | nil, should_flush: (state -> boolean) | nil}
state_times() :: %{first_item_of_batch: integer | nil, last_flush: integer | nil}
Link to this section Functions
add_item(batch_server, item) :: :ok | {:error, String.t}
Adds an item to a batch.
Returns :ok
on success, or {:error, message}
otherwise.
Forces the processing and flushing of a batch.
Returns :ok
on success, or {:error, message}
otherwise.
Link to this section Callbacks
Adds an item to the batch represented in batch
.
Returns the updated batch state or an error message.
Creates a new batch state, given the configuration options in opts
.
This function is called not just once, but every time a new batch
is created (i.e., at startup and after every flush).
Defaults to creating an empty map, returning {:ok, %{}}
.
Returns the updated batch state or an error message.
Performs some post-processing on the batch, after batch_process/1
has completed successfully. Does not return an updated batch, because
this operation is immediately followed by batch_init/1
to create a new
batch.
Returns :ok
on success, or {:error, message}
otherwise.
This is an optional callback.
Performs some pre-processing on the batch, before it is passed to
batch_process/1
. Returns the updated batch state or an error message.
This is an optional callback.
Processes the batch, whatever that entails.
This function may operate synchronously or asynchronously,
according to developer preference. If it operates synchronously,
calls to BatchPlease.flush/1
will block until finished.
Returns :ok
on success, or {:error, message}
otherwise.
Cleans up batch state before the batcher process is terminated. Defaults to no-op. This is not guaranteed to be called at termination time — for more information, see: https://hexdocs.pm/elixir/GenServer.html#c:terminate/2
Returns :ok
on success, or {:error, message}
otherwise.
This is an optional callback.
Given the current module state, returns whether the current batch
should be processed now. Precedes the handling of max_batch_size
(but does not replace it).
This is an optional callback.