Jido Runtime Architecture

This document describes the Squid Mesh runtime shape after the switchover to Jido-native coordination. It is written for new contributors and host-app maintainers who need to understand how the pieces fit together before reading individual modules.

Squid Mesh's runtime shape is:

• workflow authors keep using the Squid Mesh DSL for business workflows
• custom step modules run through Jido action contracts
• runtime coordination rebuilds from Jido-backed journals
• workflow runs and dispatch queues are represented by Jido agents
• step execution is pulled through SquidMesh.execute_next/1
• optional cron payload delivery remains backend-neutral through SquidMesh.Runtime.Runner.perform/2

If you only read three diagrams in this file, read the system overview, runtime shape, and inspection flow. The tables below are reference material for concrete boundaries.

Roadmap Alignment

The current shape is anchored in the public issue roadmap:

The ground rule from #160 still applies: the first core should prove a small, trustworthy static workflow runtime. Dynamic graph expansion, richer agent-step execution, and advanced reference workflows come after the journaled core is stable.

System Overview

flowchart TB
    subgraph HostApp[Host application]
        Workflow[Workflow modules]
        Repo[Host Repo]
        Backend[Optional lease backend]
        Workers[Worker processes]
    end

    subgraph Core[Squid Mesh core]
        API[Public API]
        Planner[Workflow planner]
        Policies[Retry, cancellation, replay, mapping]
        Inspector[Inspection and explanation]
    end

    subgraph JidoRuntime[Jido-native runtime]
        Journal[Runtime journal]
        WorkflowAgent[WorkflowAgent per run]
        DispatchAgent[DispatchAgent per queue]
        Recovery[AgentRecovery]
    end

    subgraph Jido[Jido]
        Storage[Jido.Storage]
        Actions[Jido.Action step execution]
        AgentModel[Jido.Agent state model]
    end

    Workflow --> API
    API --> Planner
    Planner --> Journal
    Journal --> Storage
    Journal --> WorkflowAgent
    Journal --> DispatchAgent
    WorkflowAgent --> DispatchAgent
    DispatchAgent --> Backend
    Backend --> Workers
    Workers --> Actions
    Actions --> DispatchAgent
    DispatchAgent --> WorkflowAgent
    WorkflowAgent --> Policies
    Policies --> Inspector
    Repo -. default Ecto storage .- Journal
    AgentModel -. agent process shape .- WorkflowAgent
    AgentModel -. agent process shape .- DispatchAgent

The key design point is that the journal, not a worker process, becomes the authority for workflow intent and dispatch lifecycle. Processes are allowed to crash and restart because their projections can be rebuilt from durable facts.

Runtime Flow

flowchart LR
    Author[Workflow author] --> DSL[Squid Mesh DSL]
    DSL --> Plan[Planner and runtime journal]
    Plan --> RunAgent[WorkflowAgent]
    RunAgent --> DispatchAgent[DispatchAgent]
    DispatchAgent --> Backend[Optional lease backend]
    Backend --> Worker[Worker loop]
    Worker --> Action[Jido action or built-in step]
    Action --> DispatchAgent
    DispatchAgent --> RunAgent
    RunAgent --> Inspect[Projection-backed inspection]
    Inspect --> Editor[SquidSonar or visual editor]

The runtime is intentionally asymmetric:

• authoring stays declarative
• durable facts stay in the journal
• execution stays in worker processes and Jido actions
• inspection stays read-only and projection-backed

Runtime Capability Matrix

flowchart LR
    Start[Start workflow run] --> RunThread[Append run_started]
    RunThread --> Plan[Append planned runnables]
    Plan --> DispatchThread[Append attempt_scheduled]
    DispatchThread --> Claim[Worker claims attempt]
    Claim --> Heartbeat[Heartbeat extends lease]
    Heartbeat --> Execute[Run Jido action]
    Execute --> Complete[Append attempt_completed or attempt_failed]
    Complete --> Apply[Append runnable_applied]
    Apply --> Decide{More runnable work?}
    Decide -- yes --> Plan
    Decide -- no --> Terminal[Append run_terminal]

The journal-backed runtime uses two different kinds of durable state:

• journal entries: the source of truth for lifecycle facts
• checkpoints: cached projections that speed up rebuilds

Checkpoints are always disposable. If a checkpoint is missing or stale, the agent can replay entries from the thread and reconstruct the same projection.

flowchart TD
    RunJournal[Run thread entries] --> RunProjection[WorkflowAgent projection]
    DispatchJournal[Dispatch thread entries] --> DispatchProjection[DispatchAgent projection]
    RunProjection --> ListRuns[list_runs]
    RunProjection --> InspectRun[inspect_run]
    RunProjection --> ExplainRun[explain_run]
    DispatchProjection --> ExplainRun
    RunProjection --> Editor[SquidSonar / visual editor]
    DispatchProjection --> Editor

This is the contract visual tooling should target. The editor reads from projections, not from worker processes or live queue internals.

Execution Ordering

The runtime is careful about which durable fact is written before the next effect becomes visible. The ordering below is the core safety model for normal step execution, retry, and successor dispatch.

sequenceDiagram
    participant API as Public API
    participant Run as Run thread
    participant Dispatch as Dispatch thread
    participant Worker as Worker loop
    participant Step as Step module

    API->>Run: append run_started
    Run->>Run: append runnable_planned
    Run->>Dispatch: append attempt_scheduled
    Worker->>Dispatch: append attempt_claimed
    Dispatch-->>Worker: claim fence
    Worker->>Step: execute input
    Step-->>Worker: ok or error
    Worker->>Dispatch: append attempt_completed or attempt_failed
    Dispatch->>Run: append runnable_applied
    Run->>Run: append successor plan or terminal fact
    Run->>Dispatch: append successor attempt if needed

The worker never becomes the authority for workflow progress. It only holds a claim fence long enough to execute a visible attempt and report the result.

Journal Threads

erDiagram
    RUN_THREAD ||--o{ RUN_ENTRY : contains
    DISPATCH_THREAD ||--o{ DISPATCH_ENTRY : contains
    RUN_INDEX_THREAD ||--o{ INDEX_ENTRY : contains
    RUN_CATALOG_THREAD ||--o{ CATALOG_ENTRY : contains
    RUN_ENTRY {
        string type
        string run_id
        string runnable_key
        string step
        datetime occurred_at
    }
    DISPATCH_ENTRY {
        string type
        string queue
        string runnable_key
        string claim_id
        datetime lease_until
    }
    INDEX_ENTRY {
        string type
        string workflow
        string queue
        string run_id
        datetime occurred_at
    }
    CATALOG_ENTRY {
        string type
        string workflow
        string queue
        string run_id
        datetime occurred_at
    }

Each append uses the current thread revision as an optimistic fence. A stale caller that tries to append based on an old projection receives a conflict instead of silently overwriting runtime state.

Projection Rebuild Map

The read model is intentionally rebuildable from journal threads. Checkpoints only shorten replay; deleting them must not change the resulting state.

flowchart TD
    subgraph Storage[Jido storage]
        RunThread[Run thread entries]
        DispatchThread[Dispatch thread entries]
        IndexThread[Run index entries]
        CatalogThread[Run catalog entries]
        Checkpoints[Projection checkpoints]
    end

    RunThread --> WorkflowAgent[WorkflowAgent projection]
    DispatchThread --> DispatchAgent[DispatchAgent projection]
    IndexThread --> RunIndex[Workflow run index]
    CatalogThread --> RunCatalog[Global run catalog]
    Checkpoints -. accelerate replay .-> WorkflowAgent
    Checkpoints -. accelerate replay .-> DispatchAgent

    WorkflowAgent --> Inspection[inspect_run snapshot]
    DispatchAgent --> Inspection
    RunIndex --> Listing[list_runs by workflow]
    RunCatalog --> Listing
    Inspection --> Explanation[explain_run details]
    Inspection --> Graph[inspect_run_graph nodes and edges]

This is the boundary SquidSonar and visual editors should depend on: listing comes from catalog or index projections, while run detail and graph views come from inspection projections.

Agents

Squid Mesh uses Jido agents as rebuildable runtime coordinators, not as a new business workflow authoring surface.

flowchart TD
    subgraph RunA[Workflow run A]
        WA[WorkflowAgent]
        RT[run thread]
        WA --- RT
    end

    subgraph Queue[Dispatch queue]
        DA[DispatchAgent]
        DT[dispatch thread]
        DA --- DT
    end

    WA -- planned runnables --> DA
    DA -- completed attempts --> WA

The phrase "a workflow run is coordinated by an agent" is useful with one important nuance: the workflow definition remains declarative data, and each step remains a business action or built-in step.

Heartbeats And Leases

Heartbeats belong to dispatch claims. They are not a second workflow state machine and they do not make external side effects exactly-once.

sequenceDiagram
    participant Worker
    participant DispatchAgent
    participant Journal
    participant WorkflowAgent

    Worker->>DispatchAgent: claim_next(owner_id)
    DispatchAgent->>Journal: append attempt_claimed(expected_rev)
    Journal-->>DispatchAgent: new thread revision
    DispatchAgent-->>Worker: claim_id + raw claim_token + lease_until

    loop while work is still active
        Worker->>DispatchAgent: heartbeat(runnable_key, claim_id, token)
        DispatchAgent->>Journal: append attempt_heartbeat(expected_rev)
        Journal-->>DispatchAgent: extended lease_until
    end

    Worker->>DispatchAgent: complete/fail with claim fence
    DispatchAgent->>Journal: append attempt_completed or attempt_failed
    DispatchAgent->>WorkflowAgent: wake/apply completed result
    WorkflowAgent->>Journal: append runnable_applied

Heartbeat rules:

For long-running steps, this heartbeat path lets the runtime distinguish "still alive" from "needs recovery". A lease-capable backend should own the concrete lease mechanics when a host needs backend-owned worker fencing, with Squid Mesh translating the resulting lifecycle facts into its dispatch projection.

Recovery Flow

stateDiagram-v2
    [*] --> RebuildWorkflowAgent
    RebuildWorkflowAgent --> RebuildDispatchAgent
    RebuildDispatchAgent --> ScheduleMissingDispatches
    ScheduleMissingDispatches --> ApplyCompletedResults
    ApplyCompletedResults --> Ready
    Ready --> [*]

    ScheduleMissingDispatches --> Conflict: stale dispatch revision
    ApplyCompletedResults --> Conflict: stale run revision
    Conflict --> RebuildWorkflowAgent

SquidMesh.Runtime.AgentRecovery drains two restart-safe windows in order:

1. Planned-but-unscheduled runnables are written to the dispatch thread.
2. Completed-but-unapplied dispatch results are written back to the run thread.

This ordering matters. A restarted node should first make all durable workflow intent visible to dispatch before applying finished work back to the workflow projection.

Failure Handling Matrix

Where Backend Leases Fit

Backend leases are optional runtime infrastructure. They are not the place where Squid Mesh workflow semantics move.

flowchart LR
    DispatchAgent --> Adapter[Lease adapter]
    Adapter --> Backend[Durable queue or lease backend]
    Backend --> Worker[Worker process]
    Worker --> Adapter
    Adapter --> DispatchAgent

This keeps setup friction low for most users while preserving an escape hatch: basic hosts can use a simple execute_next/1 worker loop, while advanced hosts can connect a backend lease adapter when they need stronger distributed worker ownership. Bedrock is the recommended reference backend today because the example app exercises queueing, delayed visibility, claims, heartbeats, completion, retry, and dead-letter behavior without coupling workflow modules to Bedrock APIs.

AI-Backed Steps

In the journal-backed runtime, the workflow run is coordinated by a WorkflowAgent. That means Squid Mesh does not need a separate step kind just because a step implementation uses an LLM, calls tools, or delegates some local decision-making to Jido.

AI-backed work should usually be modeled as an ordinary step:

step :triage_ticket, MyApp.Steps.TriageTicket,
  input: [:ticket],
  output: :triage,
  retry: [max_attempts: 2]

That keeps the important contract visible:

• the workflow owns lifecycle, retries, replay, cancellation, and audit history
• the step owns its input/output contract and side-effect safety
• model calls and tool calls stay inside the step boundary
• inspection can explain the workflow without inventing a second workflow primitive

The closed agent_step/3 issue #138 explored an explicit metadata marker for agentic steps. With the workflow run itself now coordinated by a Jido agent, that separate DSL construct is not currently part of the core runtime roadmap.

A new construct would only be worth adding later if it has different lifecycle semantics from a normal step. Examples might include a child journal, independent checkpointing, or a bounded sub-agent whose internal state must survive pause/resume, retry, replay, and deploys.

That possible shape would look like this:

flowchart TD
    ParentRun[Parent WorkflowAgent] --> AgentStep[Agent-backed step]
    AgentStep --> StepThread[Step agent journal]
    AgentStep --> ToolCalls[Tools or external services]
    AgentStep --> Human[Human approval or input]
    StepThread --> Result[Step result]
    Result --> ParentRun

Design questions before adding such a construct:

Runtime Shape

Runtime Feature Map

Projection-backed inspection rebuilds workflow and dispatch agent projections into a read-only view of pending dispatches, unapplied results, scheduled attempts, visible attempts, expired claims, manual pause or approval state, terminal state, and projection anomalies. Run-index projections rebuild workflow-scoped run lookup state from durable index entries, while the global run-catalog projection rebuilds all-run lookup state without scanning adapter internals. Both facts retain the queue each run was dispatched through and keep malformed or conflicting facts visible as anomalies. The projected explanation layer derives deterministic reason-specific details and next actions from the inspection snapshot. The public SquidMesh.inspect_run/2, SquidMesh.list_runs/2, and SquidMesh.explain_run/2 APIs expose this read model by default and infer Ecto storage from the configured repo. Host apps can still pass explicit journal_storage: or queue: overrides when a test or integration boundary needs a non-default journal boundary. Public start, listing, execution, inspection, explanation, and manual-control APIs pick up the configured defaults without repeating journal options at every call site.

The journal start path appends run, run-index, and run-catalog facts to Jido.Storage, rebuilds the workflow and dispatch agents, schedules the initial dispatch attempts from the journal, and returns the projection-backed inspection snapshot. Journal execution currently supports normal action steps, immediate built-in :log steps, built-in :wait steps in transition and dependency workflows, and manual :pause or :approval boundaries. Manual boundaries persist intervention state: unblock_run/3 resumes :pause steps, while approve_run/3 and reject_run/3 resolve :approval decisions through the configured journal runtime.

Reading Order

After this overview, read:

1. Architecture for the current component list.
2. Durable dispatch protocol for exact journal entry semantics.
3. Operations guide for current production boundaries.
4. Workflow authoring for the DSL that remains stable while backend execution choices evolve.