View Source Pachka (Pachka v1.0.0)

Pachka is a message batching library for Elixir applications.

It provides a reliable way to collect messages and deliver them in batches to configurable destinations. Messages are buffered until either the batch size limit is reached or the batch timeout occurs. Failed deliveries are automatically retried with configurable backoff strategies.

Features

  • Configurable batch sizes and timeouts
  • Customizable message sinks for different delivery targets
  • Automatic retries with customizable backoff
  • Overload protection with queue size limits
  • Graceful shutdown with message draining

Example

defmodule MyApp.MessageSink do
  @behaviour Pachka.Sink

  @impl true
  def send_batch(messages, _server_value) do
    # Process messages in batch
    :ok
  end
end

# Start Pachka server
{:ok, pid} = Pachka.start_link(
  name: MyPachka,
  sink: MyApp.MessageSink,
  max_batch_size: 100,
  max_batch_delay: :timer.seconds(1)
)

# Send messages
:ok = Pachka.send_message(MyPachka, %{event: "user_login"})
:ok = Pachka.send_message(MyPachka, %{event: "page_view"})

The messages will be collected and delivered to the sink in batches based on the configured batch size and delay parameters.

Shutdown

When a Pachka server receives a shutdown request (via Pachka.stop/2 or from its parent Supervisor), it enters termination mode. During termination, Pachka stops accepting new messages and processes all accumulated messages.

If Pachka.Sink does not implement Pachka.Sink.drain_on_terminate/3, Pachka processes accumulated messages using its standard operation flow - dividing the queue into batches and invoking Pachka.Sink.send_batch/2 for each batch. On batch failure, Pachka initiates retries with intervals determined by Pachka.Sink.retry_timeout/3.

If Pachka.Sink implements Pachka.Sink.drain_on_terminate/3, the Pachka server transfers control to this function along with all accumulated messages.

Pachka.Sink.drain_on_terminate/3

You can skip implementing Pachka.Sink.drain_on_terminate/3 when export failure risk is minimal - for instance, when storing messages in an SQLite database on the local machine's disk.

For all other scenarios, implement Pachka.Sink.drain_on_terminate/3 with a fallback system to prevent message loss during network outages or target system failures.

In-flight export

When a Pachka server enters termination mode during an ongoing export, it allows this export to complete normally within the time defined by export_timeout. This is a deliberate decision based on the notion that the intersection of two normal operations - batch export and process termination should not result in an abnormal operation - export cancellation before export_timeout expires.

If the in-flight export fails, unsent messages are returned to the queue and the entire queue is passed to Pachka.Sink.drain_on_terminate/3.

Shutdown timeout

Pachka does not define a time limit for termination mode. The actual termination time limit is set by the requesting system - either through the timeout parameter of Pachka.stop/2 or via the :shutdown option when launched through a Supervisor (see Shutdown values (:shutdown)).

Set the termination timeout based on the worst-case scenario

  • messages to send = critical_queue_size messages in queue + max_batch_size unsent messages from in-flight export
  • time to send them = shutdown timeout - export_timeout duration spent by the failed export

Startup and shutdown order

When using Pachka in your supervision tree, the order of children is important for proper startup and shutdown behavior:

  1. Start the sink system first
  2. Start Pachka servers
  3. Start message producers last

This ordering ensures:

  • During startup: The sink system is ready to receive messages before Pachka begins processing
  • During shutdown: Message producers stop first, then Pachka drains remaining messages, and finally the sink system terminates

Example configuration:

children = [
  # 1. Sink system (e.g. database connection)
  MyApp.Repo,

  # 2. Pachka server
  {Pachka, name: MyPachka, sink: MyApp.MessageSink},

  # 3. Message producers
  MyApp.UserTracker,
  MyApp.MetricsCollector
]

Supervisor.start_link(children, strategy: :one_for_one)

This order prevents message loss during system startup and shutdown since supervisors start children sequentially and shut them down in reverse order (see Start and shutdown).

Summary

Types

A message that can be sent to a Pachka server process.

Option values used by the start_link/1 function.

Options used by the start_link/1 function.

Functions

Returns a specification to start this module under a supervisor.

Sends a message to a Pachka server process.

Starts a Pachka server process linked to the current process.

Synchronously stops the server.

Types

message()

@type message() :: term()

A message that can be sent to a Pachka server process.

option()

@type option() ::
  {:sink, module()}
  | {:name, GenServer.name()}
  | {:server_value, Pachka.Sink.server_value()}
  | {:start_link_opts, GenServer.options()}
  | {:max_batch_size, pos_integer()}
  | {:critical_queue_size, pos_integer()}
  | {:max_batch_delay, timeout()}
  | {:export_timeout, timeout()}

Option values used by the start_link/1 function.

options()

@type options() :: [option()]

Options used by the start_link/1 function.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

send_message(name, message)

@spec send_message(GenServer.server(), message()) :: :ok | {:error, :overloaded}

Sends a message to a Pachka server process.

The message will be added to the queue and eventually delivered to the configured sink. If the server's queue is full, the message will be rejected.

Parameters

  • name - The name or pid of the Pachka server process
  • message - The message to be sent

Return values

Returns :ok if the message was successfully queued, or {:error, :overloaded} if the server's queue is full.

Examples

iex> Pachka.send_message(MyPachka, %{event: "user_login"})
:ok

iex> Pachka.send_message(pid, "message") 
:ok

start_link(opts)

@spec start_link(options()) :: GenServer.on_start()

Starts a Pachka server process linked to the current process.

Options

  • :sink (module/0) - Required. The module that implements the Pachka.Sink behavior for processing batched messages.

  • :name (GenServer.name/0) - The name to register the Pachka server process under. See Name Registration for details.

  • :server_value (Pachka.Sink.server_value/0) - Value passed by the server process to Pachka.Sink callback functions. Can be used to distinguish between different Pachka processes using the same sink module or to store server-specific configuration. The default value is nil.

  • :start_link_opts (GenServer.options/0) - Options passed to GenServer.start_link/3

  • :max_batch_size (pos_integer/0) - Maximum number of messages to accumulate before forcing a batch export. Batches sent to Pachka.Sink.send_batch/2 are guaranteed to be no larger than that size. The default value is 500.

  • :critical_queue_size (pos_integer/0) - Maximum number of messages the server can hold before it starts rejecting new ones. The default value is 10000.

  • :max_batch_delay (timeout/0) - Maximum time to wait before exporting a batch. The default value is 5000.

  • :export_timeout (timeout/0) - Maximum time allowed for the Pachka.Sink.send_batch/2 to process a batch. The default value is 10000.

Return values

If the server is successfully created and initialized, this function returns {:ok, pid}, where pid is the PID of the server. If a process with the specified server name already exists, this function returns {:error, {:already_started, pid}} with the PID of that process.

Raises a NimbleOptions.ValidationError exception if opts are not valid.

stop(server, timeout \\ :infinity)

@spec stop(GenServer.server(), timeout()) :: :ok

Synchronously stops the server.

Refer to the Shutdown section for information about server termination.