View Source Pachka (Pachka v1.0.0)
Pachka is a message batching library for Elixir applications.
It provides a reliable way to collect messages and deliver them in batches to configurable destinations. Messages are buffered until either the batch size limit is reached or the batch timeout occurs. Failed deliveries are automatically retried with configurable backoff strategies.
Features
- Configurable batch sizes and timeouts
- Customizable message sinks for different delivery targets
- Automatic retries with customizable backoff
- Overload protection with queue size limits
- Graceful shutdown with message draining
Example
defmodule MyApp.MessageSink do
@behaviour Pachka.Sink
@impl true
def send_batch(messages, _server_value) do
# Process messages in batch
:ok
end
end
# Start Pachka server
{:ok, pid} = Pachka.start_link(
name: MyPachka,
sink: MyApp.MessageSink,
max_batch_size: 100,
max_batch_delay: :timer.seconds(1)
)
# Send messages
:ok = Pachka.send_message(MyPachka, %{event: "user_login"})
:ok = Pachka.send_message(MyPachka, %{event: "page_view"})
The messages will be collected and delivered to the sink in batches based on the configured batch size and delay parameters.
Shutdown
When a Pachka server receives a shutdown request (via Pachka.stop/2
or from its parent Supervisor
), it enters termination mode. During termination, Pachka stops accepting new messages and processes all accumulated messages.
If Pachka.Sink
does not implement Pachka.Sink.drain_on_terminate/3
, Pachka processes accumulated messages using its standard operation flow - dividing the queue into batches and invoking Pachka.Sink.send_batch/2
for each batch. On batch failure, Pachka initiates retries with intervals determined by Pachka.Sink.retry_timeout/3
.
If Pachka.Sink
implements Pachka.Sink.drain_on_terminate/3
, the Pachka server transfers control to this function along with all accumulated messages.
Pachka.Sink.drain_on_terminate/3
You can skip implementing Pachka.Sink.drain_on_terminate/3
when export failure risk is minimal - for instance, when storing messages in an SQLite database on the local machine's disk.
For all other scenarios, implement Pachka.Sink.drain_on_terminate/3
with a fallback system to prevent message loss during network outages or target system failures.
In-flight export
When a Pachka server enters termination mode during an ongoing export, it allows this export to complete normally within the time defined by export_timeout
. This is a deliberate decision based on the notion that the intersection of two normal operations - batch export and process termination should not result in an abnormal operation - export cancellation before export_timeout
expires.
If the in-flight export fails, unsent messages are returned to the queue and the entire queue is passed to Pachka.Sink.drain_on_terminate/3
.
Shutdown timeout
Pachka does not define a time limit for termination mode. The actual termination time limit is set by the requesting system - either through the timeout
parameter of Pachka.stop/2
or via the :shutdown
option when launched through a Supervisor
(see Shutdown values (:shutdown)).
Set the termination timeout based on the worst-case scenario
- messages to send =
critical_queue_size
messages in queue +max_batch_size
unsent messages from in-flight export - time to send them =
shutdown timeout
-export_timeout
duration spent by the failed export
Startup and shutdown order
When using Pachka in your supervision tree, the order of children is important for proper startup and shutdown behavior:
- Start the sink system first
- Start Pachka servers
- Start message producers last
This ordering ensures:
- During startup: The sink system is ready to receive messages before Pachka begins processing
- During shutdown: Message producers stop first, then Pachka drains remaining messages, and finally the sink system terminates
Example configuration:
children = [
# 1. Sink system (e.g. database connection)
MyApp.Repo,
# 2. Pachka server
{Pachka, name: MyPachka, sink: MyApp.MessageSink},
# 3. Message producers
MyApp.UserTracker,
MyApp.MetricsCollector
]
Supervisor.start_link(children, strategy: :one_for_one)
This order prevents message loss during system startup and shutdown since supervisors start children sequentially and shut them down in reverse order (see Start and shutdown).
Summary
Types
@type message() :: term()
A message that can be sent to a Pachka
server process.
@type option() :: {:sink, module()} | {:name, GenServer.name()} | {:server_value, Pachka.Sink.server_value()} | {:start_link_opts, GenServer.options()} | {:max_batch_size, pos_integer()} | {:critical_queue_size, pos_integer()} | {:max_batch_delay, timeout()} | {:export_timeout, timeout()}
Option values used by the start_link/1
function.
@type options() :: [option()]
Options used by the start_link/1
function.
Functions
Returns a specification to start this module under a supervisor.
See Supervisor
.
@spec send_message(GenServer.server(), message()) :: :ok | {:error, :overloaded}
Sends a message to a Pachka
server process.
The message will be added to the queue and eventually delivered to the configured sink. If the server's queue is full, the message will be rejected.
Parameters
name
- The name or pid of the Pachka server processmessage
- The message to be sent
Return values
Returns :ok
if the message was successfully queued, or {:error, :overloaded}
if the server's queue is full.
Examples
iex> Pachka.send_message(MyPachka, %{event: "user_login"})
:ok
iex> Pachka.send_message(pid, "message")
:ok
@spec start_link(options()) :: GenServer.on_start()
Starts a Pachka
server process linked to the current process.
Options
:sink
(module/0
) - Required. The module that implements thePachka.Sink
behavior for processing batched messages.:name
(GenServer.name/0
) - The name to register the Pachka server process under. See Name Registration for details.:server_value
(Pachka.Sink.server_value/0
) - Value passed by the server process toPachka.Sink
callback functions. Can be used to distinguish between different Pachka processes using the same sink module or to store server-specific configuration. The default value isnil
.:start_link_opts
(GenServer.options/0
) - Options passed toGenServer.start_link/3
:max_batch_size
(pos_integer/0
) - Maximum number of messages to accumulate before forcing a batch export. Batches sent toPachka.Sink.send_batch/2
are guaranteed to be no larger than that size. The default value is500
.:critical_queue_size
(pos_integer/0
) - Maximum number of messages the server can hold before it starts rejecting new ones. The default value is10000
.:max_batch_delay
(timeout/0
) - Maximum time to wait before exporting a batch. The default value is5000
.:export_timeout
(timeout/0
) - Maximum time allowed for thePachka.Sink.send_batch/2
to process a batch. The default value is10000
.
Return values
If the server is successfully created and initialized, this function returns {:ok, pid}
, where pid
is the PID of the server.
If a process with the specified server name already exists, this function returns {:error, {:already_started, pid}}
with the PID of that process.
Raises a NimbleOptions.ValidationError
exception if opts
are not valid.
@spec stop(GenServer.server(), timeout()) :: :ok
Synchronously stops the server.
Refer to the Shutdown section for information about server termination.