Docket

Copy Markdown View Source

Hex Version Hex Docs CI License

Durable, graph-based workflow execution for Elixir — built for long-running, interruptible work like agentic LLM sessions, where a run may pause for an external resolution, survive a deploy, and resume exactly where it left off.

You describe a workflow as a graph document: nodes that do work, shared state fields they read and write, and edges (optionally guarded) that decide what runs next. Docket executes the graph in deterministic supersteps and commits a checkpoint at every durable transition.

Design lineage: Docket's execution model draws on Google's Pregel and LangGraph's Pregel runtime: the established bulk-synchronous pattern of computation separated by commit barriers. That model fits the BEAM naturally: Docket fans each superstep into isolated, monitored node processes, gathers their results at a mailbox barrier, and uses supervision for fault containment while PostgreSQL—not process identity—remains the recovery authority.

Features

  • Durable supersteps — runs advance in Pregel-style plan/execute/commit steps; the claim-fenced run update, schedule change, and retained events commit together in one PostgreSQL transaction.
  • External interrupts — a node parks its run as :waiting with no live process; when the outside world answers, resolve_interrupt writes the value and the run continues in the next superstep.
  • Crash-safe recovery — claim fencing admits exactly one durable winner per transition, and recovery reclaims persisted runs after a process or node failure without host-owned resume code.
  • Graphs as data — an authored graph is a plain document that round-trips through any JSON codec; publishing stores an immutable, content-addressed version, and every run records the graph ID and hash it started from.
  • Rich control flow — fan-out, fan-in barriers, guarded branches, and cycles with optional superstep bounds; node failures retry per node policy.
  • Processless testing — Docket.Test.run_inline/2 exercises full graph semantics in a unit test with no processes and no PostgreSQL.
  • Multi-tenant fairness — optional tenant scoping with claim policies that admit due work breadth-first across tenants, so one tenant cannot starve another.
  • Small core — the core depends only on telemetry; the PostgreSQL backend compiles when the host already uses ecto_sql and postgrex, and Docket takes no JSON dependency.

Docket guarantees one atomic durable winner for each committed run transition — and is explicit about where that guarantee ends. A node attempt that proposes a transition may execute more than once after a crash, timeout, or claim steal, even though only one proposal commits, so external effects need a cooperating idempotency scheme. Checkpoint observers, notifications, and telemetry are best effort. See the delivery and execution guarantees for the full matrix.

Requirements

  • Elixir 1.18+.
  • The graph core and Docket.Test have no database requirement.
  • The durable backend requires PostgreSQL 13 or newer and host-declared ecto_sql ~> 3.10 and postgrex ~> 0.17 dependencies.

Installation

The graph and execution core needs only Docket:

def deps do
  [
    {:docket, "~> 0.1.0"}
  ]
end

That is enough to build graphs and execute them processless with Docket.Test; it installs no Ecto or PostgreSQL dependency. Durable operation adds the optional database dependencies later in this guide.

Quickstart

Define a node — a module that declares its config schema and does one unit of work against the shared state:

defmodule MyApp.Nodes.ExcludeAllergens do
  @behaviour Docket.Node

  @impl true
  def config_schema do
    Docket.Schema.object(%{
      "recipes" => Docket.Schema.string(required: true),
      "avoid" => Docket.Schema.string(required: true),
      "to" => Docket.Schema.string(required: true)
    })
  end

  @impl true
  def call(state, config, _context) do
    avoid = MapSet.new(state[config["avoid"]])

    safe_recipes =
      Enum.reject(state[config["recipes"]], fn recipe ->
        Enum.any?(recipe["allergens"], &MapSet.member?(avoid, &1))
      end)

    {:ok, %{config["to"] => safe_recipes}}
  end
end

Build a graph document wiring the node between $start and $finish:

recipe_schema =
  Docket.Schema.object(%{
    "name" => Docket.Schema.string(required: true),
    "allergens" => Docket.Schema.list(:string, required: true)
  })

graph =
  Docket.Graph.new!(id: "recipe_safety")
  |> Docket.Graph.put_input!("recipes",
    schema: Docket.Schema.list(recipe_schema),
    required: true
  )
  |> Docket.Graph.put_input!("avoid", schema: {:list, :string}, required: true)
  |> Docket.Graph.put_field!("safe_recipes", schema: Docket.Schema.list(recipe_schema))
  |> Docket.Graph.put_node!("exclude_allergens",
    implementation: MyApp.Nodes.ExcludeAllergens,
    config: %{"recipes" => "recipes", "avoid" => "avoid", "to" => "safe_recipes"}
  )
  |> Docket.Graph.put_edge!("start_filter", from: "$start", to: "exclude_allergens")
  |> Docket.Graph.put_edge!("filter_finish", from: "exclude_allergens", to: "$finish")
  |> Docket.Graph.put_output!("safe_recipes", [])

Run the graph processless, with no supervision tree or database:

input = %{
  "recipes" => [
    %{"name" => "Pesto pasta", "allergens" => ["dairy", "nuts"]},
    %{"name" => "Salsa", "allergens" => []}
  ],
  "avoid" => ["dairy"]
}

{:ok, run, checkpoints} = Docket.Test.run_inline(graph, input)

run.status  #=> :done
run.output  #=> %{"safe_recipes" => [%{"name" => "Salsa", "allergens" => []}]}
Enum.map(checkpoints, & &1.type)
#=> [:run_initialized, :step_committed, :run_completed]

Durable PostgreSQL

PostgreSQL support is opt-in. Docket does not install Ecto or Postgrex for core-only applications. The application that owns the Repo should declare all three dependencies directly:

def deps do
  [
    {:docket, "~> 0.1.0"},
    {:ecto_sql, "~> 3.10"},
    {:postgrex, "~> 0.17"}
  ]
end

Install the tables through a host-owned migration:

mix deps.get
mix docket.gen.migration -r MyApp.Repo
mix ecto.migrate -r MyApp.Repo

Configure a tenantless, poll-only runtime with explicit retention:

defmodule MyApp.Docket do
  use Docket,
    tenant_mode: :none,
    backend:
      {Docket.Postgres,
       repo: MyApp.Repo,
       notifier: :none,
       pruner: [
         interval_ms: :timer.hours(1),
         event_retention_ms: :timer.hours(24 * 30),
         run_retention_ms: :timer.hours(24 * 90),
         batch_size: 1_000
       ]}
end

children = [MyApp.Repo, MyApp.Docket]
Supervisor.start_link(children, strategy: :one_for_one)

The generated migration delegates to Docket's pinned schema version; commit it with the application rather than running migrations at application startup.

Publish an immutable graph version, then start durable work from the returned reference:

{:ok, graph_ref} = MyApp.Docket.save_graph(graph)

{:ok, run} = MyApp.Docket.start_run(graph_ref, input)

{:ok, finished} = MyApp.Docket.await_run(run.id, timeout: 5_000)
finished.status #=> :done
finished.output #=> run.output

start_run returns after the initialized run is durably committed; production advancement is asynchronous, and await_run is a bounded convenience for callers that need to wait until a run pauses or terminates. The facade also provides paged run, event, and graph-version readers plus fetch_run, inspect_run, cancel_run, and retry_poisoned_run; the Docket module docs are the authoritative API reference.

Multi-tenant applications configure tenant_mode: :required with a fair claim policy such as Docket.Postgres.ClaimPolicy.WindowedInterleave, and durable integration tests use the production lifecycle through testing: :inline or testing: :manual. See the parent-application example and the PostgreSQL operations guide.

External interrupts

A node pauses the run by returning an interrupt naming the state field where the external resolution should be written:

def call(state, _config, _context) do
  case state["decision"] do
    nil -> {:interrupt, %Docket.Interrupt{resume_channel: "decision"}}
    decision -> {:ok, %{"applied" => decision}}
  end
end

The run commits as :waiting without retaining a per-run process. When the external system resolves it:

{:ok, run} =
  MyApp.Docket.resolve_interrupt(run_id, interrupt_id, "approved")

The value is validated against the interrupt's schema (if any), written to the resume field, and the interrupted node re-executes in the next superstep with the resolved value visible in its state.

Execution model

A run advances in Pregel-style supersteps:

  1. Plan — from committed state only, select the activated nodes and build their task descriptors (deterministic IDs and idempotency keys).
  2. Execute — dispatch each activation through the configured executor. Nodes see the same committed snapshot; nothing observes a same-step write.
  3. Commit — validate writes against field schemas, apply reducers, resolve same-step conflicts in sorted node order, evaluate edge guards and fan-in barriers, and commit the step atomically with a checkpoint.

Current boundaries

Docket deliberately leaves these to the host application:

  • Authorization and ownership checks before calling a Docket facade.
  • Graph publish workflows and application-level version selection.
  • UI projections for editors and live run views.
  • External effects — nodes call your code; Docket never talks to the network.

The configured backend exclusively owns graph/run persistence, scheduling, recovery, signals, and production supervision.

Learn more