Zizq.BatchConfig (Zizq v0.6.1)

Copy Markdown View Source

Batching configuration: fold an enqueue into an existing :ready job rather than creating a new one.

Say where the batch accumulates and how large it may get, and the rest follows:

# Cap `.device_ids` at 100 entries per batch.
Zizq.BatchConfig.new!(limit: 100, path: ".device_ids")

# The whole payload is the batch, and is an array.
Zizq.BatchConfig.new!(limit: 1_000)

Declared on a job module, an enqueue then carries only its own contribution and the server merges it:

defmodule MyApp.PushBatch do
  use Zizq.JobKind,
    type: "push",
    queue: "push",
    batch: [limit: 100, path: ".device_ids"]

  @impl Zizq.JobKind
  def perform(%{"device_ids" => ids}), do: MyApp.Push.deliver(ids)
end

MyApp.PushBatch.new(%{"device_ids" => [id], "platform" => "apple"})
|> Zizq.enqueue(MyApp.Zizq)

What gets generated

OptionBecomes
:limit and :paththe :when predicate
:path, :dedup, :sortedthe :fold expression
:paththe default :key, hashing everything except the batch

:key defaulting to the payload minus the batch path is what makes the common case work without saying anything: two enqueues alike in every respect but what they contribute belong in the same batch. In the example above, platform decides the batch and device_ids accumulates into it.

:dedup folds with jq's unique, :sorted with sort. unique also sorts, so :dedup subsumes :sorted.

Options

  • :limit — maximum combined length at :path before the batch is sealed and a new one starts.
  • :path — jq path to the value that accumulates. Defaults to ".", the whole payload.
  • :key — override the derived key with a string, or another Zizq.PayloadHasher.
  • :dedup — fold with unique.
  • :sorted — fold with sort.

Writing the expressions by hand

:key, :when and :fold can be given directly instead, for folds the templates do not cover — accumulating into a map, say, or counting rather than appending:

Zizq.BatchConfig.new!(
  key: "digest:tenant-42",
  when: "$existing.count < 100",
  fold: "$existing | .count += 1 | .ids += $new.ids"
)

Both are jq, evaluated with $existing bound to the batch's current payload and $new to the incoming one. :limit and :path cannot be mixed with them, since they would be generating the same fields.

Batching is a Pro-licensed feature; without one the server responds 403, which surfaces as %Zizq.Error{reason: :forbidden}.

The configuration is returned on job reads, and the first enqueue's :when and :fold govern the whole batch — so reading back what is stored is how a "first wins" surprise gets diagnosed.

Summary

Functions

Build a batch configuration from a keyword list or map.

Types

t()

@type t() :: %Zizq.BatchConfig{
  fold: String.t(),
  key: String.t() | Zizq.PayloadHasher.t(),
  when: String.t()
}

Functions

new!(batch)

@spec new!(t() | keyword() | map()) :: t()

Build a batch configuration from a keyword list or map.

See the module documentation for the options.