Projection Internals

Copy Markdown View Source

Jidoka.Projection is the single dispatch surface that converts every Jidoka data contract into a stable, compact map. Projections are deliberately smaller than raw structs: they omit Zoi schemas, full LLMDB.Model structs, Spark module metadata, and other implementation-private values. They are the contract shared by Jidoka.inspect/2, golden tests, trace sinks, replay scaffolding, and UI consumers like Jidoka.AgentView. This guide walks the dispatch table, explains why projections look the way they do, and gives contributors the rules for adding or changing a projection without breaking external consumers. It is written for people maintaining Jidoka.Projection, Jidoka.Inspection, and Jidoka.AgentView, not for agent authors.

When To Use This

  • Use this guide before adding or modifying a project/1 clause.
  • Use this guide when introducing a new Jidoka data struct that should be inspectable (it needs a project/1 clause, often a matching Jidoka.Inspection.inspect/2 clause, and golden coverage).
  • Use this guide when removing a field from a struct, because consumers may depend on the projection key being present.
  • Do not use this guide as a tutorial on debugging an agent. Authors should read Inspection And Preflight for the user-facing surface.

Prerequisites

  • Elixir ~> 1.18 and a checkout of the jidoka package.
  • Familiarity with the structs in lib/jidoka/agent.ex, lib/jidoka/turn/, and lib/jidoka/effect/.
  • Awareness that golden tests in test/jidoka/golden/ snapshot projection output verbatim.
mix deps.get
mix test test/jidoka/projection_test.exs
mix test test/jidoka/golden/
mix test test/jidoka/inspection_test.exs

Quick Example

Projection is invoked through Jidoka.project/1. The result is a map (or a list of maps) suitable for assertions, JSON encoding, and UI rendering:

spec = Jidoka.agent!(id: "demo", model: %{provider: :test, id: "m"})

Jidoka.project(spec)
#=> %{
#     id: "demo",
#     model: "test:m",
#     instructions: "...",
#     context_schema?: false,
#     result: nil,
#     memory: nil,
#     operations: [],
#     controls: %{max_turns: nil, timeout_ms: nil, inputs: [], outputs: [], operations: [], metadata: %{}},
#     runtime_defaults: %{},
#     metadata: %{...}
#   }

Compare that with Jidoka.inspect(spec), which adds derived views (kind, module name when known, the plan, the timeline, the journal):

Jidoka.inspect(spec)
#=> %{kind: :agent, module: nil, spec: %{...}, plan: %{...}}

Both functions are pure; both are golden-tested.

Concepts

Three ideas explain the projection contract.

  1. project/1 is the machine-readable form. It is what tests assert on, what trace sinks serialize, and what UIs render. Output must be plain Elixir data (maps, lists, strings, atoms, numbers, booleans, nil).
  2. Jidoka.Inspection.inspect/2 is the human-readable form. It composes projections into named "views" (:agent, :turn, :turn_state, :snapshot, :session, :replay, :effect_journal, :effect_intent, :effect_result, :review_*, :memory_*, :eval_run). Views always include a :kind key so consumers can dispatch.
  3. Projections shrink struct payloads on purpose. Removing Zoi schemas, LLMDB.Model internals, Spark metadata, and unstable nested structures is what makes the contract stable across implementation churn.
                  Jidoka data structs
                          │
            ╭─────────────┴────────────────╮
            ▼                              ▼
     Jidoka.project/1            Jidoka.Inspection.inspect/2
            │                              │
            ▼                              ▼
   plain data maps                    named view maps
   (lists of maps)                    (with :kind key)
            │                              │
   ╭────────┴───────╮            ╭─────────┼───────────╮
   ▼                ▼            ▼         ▼           ▼
golden tests   trace sinks   debug logs UI/CLI   Kino/Livebook
                replay         eval     widgets    cells
                consumers      runs

How To

Step 1: Read The Dispatch Table

Jidoka.Projection is one screen per supported struct. The pattern is always the same:

def project(%Agent.Spec{} = spec) do
  %{
    id: spec.id,
    instructions: spec.instructions,
    model: Jidoka.Config.model_ref(spec.model),
    generation: project(spec.generation),
    context_schema?: not is_nil(spec.context_schema),
    result: project(spec.result),
    memory: project(spec.memory),
    operations: Enum.map(spec.operations, &project/1),
    controls: project(spec.controls),
    runtime_defaults: project_value(spec.runtime_defaults),
    metadata: project_agent_metadata(spec.metadata)
  }
end

Three rules apply to every clause:

  • Composition over flattening. A struct's nested struct fields go through project/1 again; lists of structs go through Enum.map(&project/1).
  • project_value/1 is the catch-all. Any value that does not have a clause is funneled through project_value/1, which strips known unstable values (Zoi schemas, LLMDB.Model, exceptions) and walks maps/lists recursively.
  • Booleans answer "is there one?" Fields like context_schema and result.schema are reduced to context_schema? and schema? booleans, because the schema itself is opaque.

Step 2: Strip Unstable Values With project_value/1

project_value/1 is the only place where struct-aware stripping happens:

defp project_value(%_{} = exception) when is_exception(exception), do: Error.to_map(exception)

defp project_value(%LLMDB.Model{} = model), do: Jidoka.Config.model_ref(model)

defp project_value(%module{} = struct) do
  if zoi_schema?(module) do
    %{schema?: true}
  else
    struct
    |> Map.from_struct()
    |> project_value()
  end
end

defp project_value(%{} = map) do
  Map.new(map, fn {key, value} -> {key, project_value(value)} end)
end

defp project_value(list) when is_list(list), do: Enum.map(list, &project_value/1)
defp project_value(value), do: value

Three behaviors to remember:

  • Exceptions become maps. Error.to_map/1 sanitizes credential-shaped values and returns a flat representation.
  • Zoi schemas become %{schema?: true}. Schemas are huge nested structs that change shape with Zoi version bumps. The boolean is the stable form.
  • Foreign structs are deep-mapped. A struct without a dedicated project/1 clause is flattened to a plain map first, then projected recursively. Use this sparingly; named clauses are better.

Step 3: Strip Author Metadata From Specs

project_agent_metadata/1 and project_operation_metadata/1 are spec-specific cleaners:

defp project_agent_metadata(metadata) when is_map(metadata) do
  metadata
  |> Map.drop(["dsl_module", :dsl_module])
  |> project_value()
end

defp project_operation_metadata(metadata) when is_map(metadata) do
  has_parameters_schema? =
    is_map(Map.get(metadata, "parameters_schema") || Map.get(metadata, :parameters_schema))

  metadata
  |> Map.drop(["parameters_schema", :parameters_schema])
  |> project_value()
  |> Map.put("parameters_schema?", has_parameters_schema?)
end

Two rules:

  • DSL module references are dropped. They are runtime-bound; including them in golden tests pins the test to a specific module name.
  • Parameter schemas become booleans. The full schema map is meaningful for Jido but noisy for golden tests; the "parameters_schema?" flag is stable.

Step 4: Build A Named View

Jidoka.Inspection is the second layer. It composes projections into named views:

defp turn_result_view(%Turn.Result{} = result) do
  %{
    kind: :turn,
    status: :finished,
    content: result.content,
    timeline: timeline(result.events),
    journal: Jidoka.project(result.journal),
    result: Jidoka.project(result)
  }
end

Three conventions:

  • Every view has a :kind key. It is the dispatch field for consumers that see a mix of view types (for example, a UI widget that toggles between turn results and snapshots).
  • The :timeline field uses Jidoka.Trace.timeline/1. That function shrinks raw events into trace-shaped maps; UIs and tests should prefer it over raw events.
  • The original projection is always included. Views are additive; they never drop fields from project/1.

Step 5: Read The Preflight Struct

Jidoka.Inspection.Preflight is the struct returned by Jidoka.preflight/3. It is itself defined as a Zoi-backed struct so that preflight output is also data:

@schema Zoi.struct(
          __MODULE__,
          %{
            agent: Zoi.map(),
            plan: Zoi.map(),
            request: Zoi.map(),
            prompt: Zoi.map(),
            events: Zoi.array(Zoi.map()) |> Zoi.default([]),
            timeline: Zoi.array(Zoi.map()) |> Zoi.default([]),
            diagnostics: Zoi.array(Zoi.any()) |> Zoi.default([])
          },
          coerce: true
        )

Preflight is produced by Jidoka.Inspection.preflight/3, which resolves a plan, normalizes a request, runs the pure Jidoka.Workflow.Steps.assemble_prompt/1, and projects the resulting state. No capability is called. The struct is the contract for "what would a turn see?" debugging without spending a token.

Step 6: Use The AgentView Projection

Jidoka.AgentView is a Zoi-backed struct intended for UI consumers. It is projection-only: no pid, no provider client, no persistence. The struct carries visible_messages, streaming_message, events, status, outcome, and a metadata slot that can hold an agent_state reference and the last result projection.

AgentView.after_turn/2 is the main reduction:

def after_turn(%__MODULE__{} = view, {:ok, %Turn.Result{} = result}) do
  %{
    view
    | visible_messages: commit_pending(view.visible_messages) ++ [assistant_message(result.content)],
      streaming_message: nil,
      events: append_operation_events(view.events, result),
      status: :idle,
      outcome: {:ok, result},
      metadata:
        view.metadata
        |> Map.put(:agent_state, result.agent_state)
        |> Map.put(:last_result, Jidoka.project(result))
  }
end

Two rules contributors must keep:

  • Anything UI consumers see is a projection or a plain map. Never expose a raw Turn.Result field directly through AgentView.
  • Streaming deltas update streaming_message; non-delta events go into events. That separation is what lets LiveView widgets render incrementally without keeping the full event log in DOM.

Step 7: Maintain Golden Coverage

Golden tests live under test/jidoka/golden/ and pin the projection output verbatim. The pattern is:

assert Jidoka.project(MinimalAgent.spec()) == %{
         id: "golden_minimal_agent",
         instructions: Jidoka.Agent.default_instructions(),
         model: "test:golden-minimal-model",
         generation: %{params: %{...}, provider_options: %{}, extra: %{}},
         context_schema?: false,
         result: nil,
         memory: nil,
         operations: [],
         controls: %{...},
         runtime_defaults: %{},
         metadata: %{...}
       }

Any change to the projection of a struct must update the matching golden expectations in the same commit. Skipping that step makes the test fail and hides the real change in noise.

Common Patterns

  • Add a project/1 clause whenever you add a Zoi-backed struct that carries durable data. Skipping the clause forces consumers into the catch-all project_value/1, which is unstable.
  • Use Enum.reject/2 to drop nil-valued keys on small structs. The pattern shows up in OperationResult and RecallResult: nil keys produce noisy golden output.
  • Prefer named views over ad-hoc maps. If a struct has more than one consumer, add it to Jidoka.Inspection.inspect/2 so the :kind dispatch works.
  • Always include the original projection inside a view. A view that omits the underlying projection forces consumers to call Jidoka.project/1 again separately.

Change Points

  • New project/1 clauses. The struct must be a Zoi-backed struct or a plain Elixir map; functions, pids, and refs are rejected.
  • New named views. Add a clause to Jidoka.Inspection.inspect/2 and a matching private helper that returns a map with :kind.
  • Custom unstable value handling. Add a clause to project_value/1 before the generic %module{} = struct clause. Keep the new clause tight (one struct, one rewrite).
  • AgentView derivations. UI-specific reductions belong inside AgentView. Avoid adding UI-only fields to a projected struct.

Invariants

  1. Projections are plain Elixir data. No structs in the output except inside result.value (which is application-defined and projected through project_value/1).
  2. project/1 is total. Every struct that escapes a Jidoka API call must have either a dedicated clause or a stable project_value/1 reduction.
  3. Zoi schemas never leak. They are reduced to %{schema?: true} or to booleans like context_schema?.
  4. LLMDB.Model becomes a string. Jidoka.Config.model_ref/1 produces "provider:id". Embedding the full model struct in a projection is a bug.
  5. Spark DSL module references are stripped from spec metadata. The dsl_module key is dropped so golden tests do not pin a module name.
  6. Inspection views always include :kind. Consumers depend on it to dispatch.
  7. Preflight is effect-free. Adding a clause that calls a capability from inside Inspection.preflight/3 breaks the contract.
  8. AgentView carries no live values. Pids, sockets, provider clients, and supervisor references are never assigned to AgentView fields.

Testing

The two key surfaces are test/jidoka/projection_test.exs for clause behavior and test/jidoka/golden/ for pinned output. Golden tests are the guardrail; project tests assert smaller properties.

test "operation projection drops parameters_schema struct but keeps boolean" do
  operation =
    Jidoka.Agent.Spec.Operation.new!(
      name: "demo",
      description: "demo",
      idempotency: :idempotent,
      metadata: %{"parameters_schema" => %{type: "object"}}
    )

  projected = Jidoka.project(operation)

  refute Map.has_key?(projected.metadata, "parameters_schema")
  assert projected.metadata["parameters_schema?"] == true
end

For inspection, prefer asserting the :kind plus a small projection:

test "inspect/2 of a turn result has kind :turn and content" do
  result = build_turn_result()
  view = Jidoka.inspect(result)
  assert view.kind == :turn
  assert view.content == result.content
  assert is_list(view.timeline)
end

Troubleshooting

SymptomLikely CauseFix
Golden test fails after adding a struct fieldProjection grew or shrankUpdate the matching golden assertion in the same commit.
Projection contains %Zoi.Types.Object{...}A Zoi schema escaped project_value/1Add an explicit clause that reduces it to %{schema?: true} or a boolean.
Projection contains a function or pidA runtime value leaked into struct fieldsMove the value into a capability and remove it from the struct, or store an opaque id instead.
Jidoka.inspect(my_struct) returns the raw struct mapNo matching clause in Jidoka.Inspection.inspect/2Add a named view clause and a helper that returns %{kind: :my_struct, ...}.
Jidoka.preflight/3 errors with :invalid_agent_moduleModule passed is not a Jidoka.Agent DSL modulePass a Jidoka.Agent.Spec, a Jidoka.Turn.Plan, or a module that exports spec/0.
AgentView shows wrong content after a turnafter_turn/2 did not update streaming_message to nilAlways reset streaming_message: nil in after_turn/2 clauses.
Trace timeline empty for a known turnEvents list passed to Trace.timeline/1 was empty (turn errored before any event)Use Turn.Result.events from a successful turn; failed turns still emit :turn_failed.

Reference