Human In The Loop

Copy Markdown View Source

Jidoka treats human review as a durable pause, not as a callback or a side-channel. An operation control returns {:interrupt, reason} before the operation runs, the turn hibernates at a review cursor, and an application later resumes the same snapshot with an approval or a denial.

When To Use This

  • Use durable approvals for any action a model can request that you do not want to execute automatically: refunds, deletes, deploys, sends to external systems.
  • Use durable approvals for compliance flows where the reviewer is a different process or even a different deployment than the runtime.
  • Do not use durable approvals as a substitute for input/output controls. Those run at different boundaries; see Controls.

Prerequisites

  • A Jidoka agent with at least one operation whose call you want to gate.
  • A persistent session, or a place to store the hibernation snapshot. See Sessions And Stores and Snapshots And Resume.
  • Familiarity with the operation control surface in Controls.
mix deps.get
mix test

Quick Example

Mark the operation as requiring approval. Jidoka pauses before the operation executes, returns a snapshot, and resumes after the application approves or denies the review request.

defmodule MyApp.SupportAgent do
  use Jidoka.Agent

  agent :support_agent do
    instructions "Use refund_order when the customer asks for a refund."
  end

  tools do
    action MyApp.RefundOrder,
      idempotency: :unsafe_once,
      approval: [
        reason: :refund_requires_review,
        message: "Review the refund before it is issued.",
        ttl_ms: 300_000
      ]
  end
end

llm = fn _intent, journal, _ctx ->
  case map_size(journal.results) do
    0 -> {:ok, %{type: :operation, name: "refund_order",
                 arguments: %{"order_id" => "A1001"}}}
    _ -> {:ok, %{type: :final, content: "Refunded A1001."}}
  end
end

operations = fn _intent, _journal, _ctx -> {:ok, %{refunded: true}} end

{:hibernate, snapshot} =
  Jidoka.turn(MyApp.SupportAgent, "Refund A1001",
    llm: llm,
    operations: operations
  )

{:ok, [review]} = Jidoka.pending_reviews(snapshot)

{:ok, %Jidoka.Turn.Result{content: "Refunded A1001."}} =
  Jidoka.approve(snapshot, review,
    llm: llm,
    operations: operations
  )

Concepts

The data path is simple. Every stage is just data.

╭───────────────────────╮     ╭──────────────────────╮
│ Operation control     │────▶│ {:interrupt, reason} │
╰───────────────────────╯     ╰──────┬───────────────╯
                                     │
                                     ▼
                          ╭──────────────────────╮
                          │ Review.Interrupt     │
                          │ snapshot.cursor=:review │
                          ╰──────┬───────────────╯
                                 │
                                 ▼
                          ╭──────────────────────╮
                          │ Review.Request       │
                          │ (snapshot.metadata)  │
                          ╰──────┬───────────────╯
                                 │
                                 ▼
                          ╭──────────────────────╮
                          │ Review.Response      │
                          │ approve / deny       │
                          ╰──────┬───────────────╯
                                 │
                                 ▼
                          ╭──────────────────────╮
                          │ Jidoka.resume/2      │
                          │ approval: response   │
                          ╰──────────────────────╯
  • Jidoka.Review.Interrupt is the runtime pause. It records the agent, request, effect id, operation, arguments, idempotency, and an optional expires_at_ms.
  • Jidoka.Review.Request is the application-facing view. It is built from the interrupt and stored on snapshot.metadata["pending_review"] and session.pending_reviews.
  • Jidoka.Review.Response is the small struct the application creates to resume: either :approved or :denied, always targeting one interrupt id.
  • The interrupt only fires for :operation boundaries today; input and output controls cannot durably hibernate yet and must block or fail.
  • Tool-level approval: compiles to Jidoka's built-in Jidoka.Controls.RequireApproval operation control. Approval predicates and custom operation controls use the same interrupt, snapshot, and resume path.

How To

Step 1: Add Approval Policy

Use approval: when the policy is unconditional, can be expressed with only: / except: filters, or can be delegated to a module predicate:

tools do
  action MyApp.RefundOrder,
    idempotency: :unsafe_once,
    approval: true
end

When approval depends on operation arguments or caller context, use a module-based predicate. The predicate receives Jidoka.Context:

defmodule MyApp.LargeRefundPredicate do
  use Jidoka.ApprovalPredicate

  @impl true
  def call(%Jidoka.Context{} = ctx) do
    amount = Map.get(ctx.arguments, "amount") || 0
    amount >= 100 or Jidoka.Context.get(ctx, :tenant_id) == "enterprise"
  end
end

Attach it to the tool's approval policy:

tools do
  action MyApp.RefundOrder,
    idempotency: :unsafe_once,
    approval: [
      when: MyApp.LargeRefundPredicate,
      reason: :large_refund_review
    ]
end

Request-level approval is useful when the caller wants one reviewed run without changing the agent spec:

Jidoka.turn(MyApp.SupportAgent, "Refund A1001",
  require_tool_approval: [only: ["refund_order"]]
)

Define a custom operation control when the policy needs to block, interrupt with a custom shape, or do something broader than standard approval.

Operation controls receive a Jidoka.Runtime.Controls.OperationContext. Returning {:interrupt, reason} flips the turn into the review path.

defmodule MyApp.RequireRefundApproval do
  use Jidoka.Control, name: "require_refund_approval"

  @impl true
  def call(%Jidoka.Runtime.Controls.OperationContext{} = operation) do
    if operation.operation == "refund_order" do
      {:interrupt, :approval_required}
    else
      :cont
    end
  end
end

Match the control as narrowly as possible. Common keys are kind, name, source, idempotency, and any metadata. Risky operations declared with idempotency: :unsafe_once must have approval or an operation control before the spec can compile. See Idempotency And Safety.

Step 2: Observe The Hibernation

Running the turn returns {:hibernate, snapshot} instead of {:ok, result}. Read the pending review off the snapshot.

{:hibernate, snapshot} =
  Jidoka.turn(MyApp.SupportAgent, "Refund A1001",
    llm: llm,
    operations: operations
  )

snapshot.cursor.phase
#=> :review

%Jidoka.Review.Request{} = review = snapshot.metadata["pending_review"]
review.operation
#=> "refund_order"
review.arguments
#=> %{"order_id" => "A1001"}

When the turn runs inside a session, the same request is mirrored on the session struct:

{:ok, [^review]} = Jidoka.Session.pending_reviews(session)

Step 3: Approve And Resume

Jidoka.approve/3 approves a pending review and resumes the snapshot:

{:ok, [review]} = Jidoka.pending_reviews(snapshot)

{:ok, %Jidoka.Turn.Result{}} =
  Jidoka.approve(snapshot,
    review,
    llm: llm,
    operations: operations
  )

Use the lower-level response API when you want to persist or transform the approval response explicitly. Jidoka.Review.Response.approve/2 builds the response targeted at the pending interrupt.

approval = Jidoka.Review.Response.approve(review.interrupt_id)

{:ok, %Jidoka.Turn.Result{}} =
  Jidoka.resume(snapshot,
    approval: approval,
    llm: llm,
    operations: operations
  )

The runner does not re-run the operation control for the approved interrupt. The journal still prevents duplicate effects on later resumes: once the operation result is in the journal, replaying the snapshot reuses the recorded result instead of calling the capability again.

Step 4: Deny For A Deterministic Non-Execution

A denial resumes the same snapshot but never calls the operation capability.

{:error, reason} =
  Jidoka.deny(snapshot,
    review,
    reason: :policy_denied,
    llm: llm,
    operations: operations
  )

Jidoka.deny/3 still returns the normal resume error shape for denied approvals. Surface the response details to the caller; they carry the reason, responded_at_ms, and metadata the reviewer attached.

Step 5: Honor Expiration Windows

Set ttl_ms: on the approval policy when a tool has its own review window. Pass :approval_ttl_ms at runtime when the caller should override that window for one turn. The runtime stamps expires_at_ms on the interrupt at hibernation time.

{:hibernate, snapshot} =
  Jidoka.turn(MyApp.SupportAgent, "Refund A1001",
    llm: llm,
    operations: operations,
    approval_ttl_ms: 60_000
  )

review = snapshot.metadata["pending_review"]
review.expires_at_ms
#=> 1717250000000

When resume runs after expiry, validation rejects the response:

late = Jidoka.Review.Response.approve(review.interrupt_id,
  responded_at_ms: review.expires_at_ms + 1)

{:error, {:approval_expired, _id, _now, _expires}} =
  Jidoka.resume(snapshot,
    approval: late,
    llm: llm,
    operations: operations
  )

Application code should set responded_at_ms from the same clock it uses for created_at_ms; the runtime fills it in with the current system clock if you leave it nil.

Step 6: List Pending Reviews Across A Store

A session keeps its own pending requests. A store can flatten across sessions for an operator dashboard.

{:ok, [_review | _rest]} = Jidoka.Session.pending_reviews(store)

Each entry is a Jidoka.Review.Request struct. It carries everything an operator needs to render the decision: agent id, operation, arguments, reason, and the expiration timestamp.

Common Patterns

  • Use approval sugar first. Reach for approval: true or approval: [only: ...] before writing a custom control.
  • One control per risk class. When approval needs code, keep the control logic to a simple match on operation and kind. Push richer policy into the application-side approval workflow.
  • Persist before showing to a reviewer. Always save the snapshot (or its session) before exposing the pending review. The reviewer's decision must be able to find the same snapshot later.
  • Pass :approval_ttl_ms. Even a long TTL is safer than none. Expired approvals fail deterministically.
  • Treat denials as expected. A {:error, {:approval_denied, _}} return value should be logged but is not a runtime fault.
  • Do not retry approvals. Once an interrupt is resolved its effect result is journaled. A second Jidoka.resume/2 against the same snapshot will reuse that result, not call the operation again.

Testing

Use deterministic fakes for both the LLM and the operations capability.

test "approval resumes the pending refund" do
  llm = fn _intent, journal, _ctx ->
    case map_size(journal.results) do
      0 -> {:ok, %{type: :operation, name: "refund_order",
                   arguments: %{"order_id" => "A1001"}}}
      _ -> {:ok, %{type: :final, content: "Refunded A1001."}}
    end
  end

  operations = fn _intent, _journal, _ctx -> {:ok, %{refunded: true}} end

  assert {:hibernate, snapshot} =
           Jidoka.turn(MyApp.SupportAgent, "Refund A1001",
             llm: llm,
             operations: operations
           )

  review = snapshot.metadata["pending_review"]
  approval = Jidoka.Review.Response.approve(review.interrupt_id)

  assert {:ok, %Jidoka.Turn.Result{content: "Refunded A1001."}} =
           Jidoka.resume(snapshot,
             approval: approval,
             llm: llm,
             operations: operations
           )
end

A denial test mirrors the approval test: build the snapshot, call Jidoka.Review.Response.deny/2, assert on {:error, {:approval_denied, _response}}, and assert the operations capability was never invoked.

Troubleshooting

SymptomLikely CauseFix
{:error, {:approval_interrupt_mismatch, expected, actual}}Approval was built for a different interrupt id.Always read review.interrupt_id from the live snapshot.
{:error, {:approval_expired, _, _, _}}Response came in after expires_at_ms.Build a fresh review request, or extend :approval_ttl_ms.
{:error, {:approval_denied, response}}Reviewer denied the action.Surface the response to the caller; do not retry.
Operation is called twice after approvalSnapshot was resumed twice without checking the result.Resume each snapshot once; persist Turn.Result after success.
{:error, {:unsafe_once_requires_control, name, kind}} at compileAn :unsafe_once operation has no approval policy or operation control.Add approval: true or a matching operation control before compiling the plan.
snapshot.metadata["pending_review"] is nilThe hibernation came from :after_prompt/:before_each_effect, not a review.Inspect snapshot.cursor.phase; only :review produces a pending request.

Reference

Key modules touched in this guide: