# The Flow Compiler Baton has two ways to build a DAG. You can assemble one in code with `Baton.new/1` and `Baton.add/4` (see the [building a workflow guide](building_a_workflow.md)), or you can **compile one from data**. This guide covers the second path: `Baton.Flow.Compiler` takes a portable, JSON-serializable *definition* and turns it into an executable Baton workflow. One idea drives the whole design: **the persisted artifact is inert data.** A definition contains no executable code and names no host module. Everything live is resolved later, through host-supplied seams. ``` Definition Compiler Baton workflow (JSON data) ─────▶ validate (Oban jobs) load context ─────▶ ┌──────────┐ expand │ search │ build └────┬─────┘ snapshot ▼ ┌──────────┐ │ analyze │ … └──────────┘ ``` ## 1. The portable definition A definition is a struct of plain JSON values. `Baton.Flow.Definition` handles `load/1`, `decode/1`, `dump/1`, and `encode/1`, so a host may store it in Ecto, Git, a file, or another service and hand the decoded form to Baton. ``` Definition ─┬─ format wire version, currently 1 ├─ key, name ├─ metadata e.g. %{"context_provider" => "document"} └─ nodes: [ NodeSpec ] ├─ id, type "llm" | "action" ├─ deps [logical node ids] ├─ config prompts, bindings, model, action key └─ fan_out: FanOutSpec | nil ├─ collection $input.… / $context.… ├─ item_id $item.… └─ gate parallel | sequential ``` `Baton.Flow.NodeSpec` deliberately holds no module references — runtime modules are resolved from a host registry much later, keyed by the node's `type`, `action`, or `adapter` string. ### Bindings `Baton.Flow.Binding` is the entire expression language: a `$`-prefixed dotted path rooted at one of five roots. | Root | Resolves to | | --- | --- | | `$input.…` | the portable input passed to `compile/2` | | `$context.…` | domain context loaded by the host | | `$steps.…` | a completed upstream step's stored result — a list of them if it fanned out | | `$run.…` | `workflow_id`, `definition_ref` | | `$item.…` | the current item, inside a fan-out expansion | Maps and lists are traversed recursively, so a node can declare an entire input object without embedding code in its persisted definition. An LLM node's `model` takes a binding too, not just a literal id. `$item.model` gives each node of a fan-out its own model; `$input.model` lets one definition run under a caller-chosen one. Whatever it resolves to must be a string, and a `$steps.…` model is held to the same declared-dep rule as any other expression in the node's config. ## 2. The compile pipeline `Baton.Flow.Compiler.compile/2` runs five stages, short-circuiting on the first error. ``` Definition + opts (:input, :context, :definition_ref, :name) │ ▼ ┌───────────────────────────────────────────────────────────┐ │ 1. VALIDATE Baton.Flow.Validator.validate/1 │ │ node types ∈ {llm, action} │ │ fan-out shape: collection is $input./$context. or │ │ $steps.; item_id is │ │ $item.; gate is known; max_items ≥ 1 │ │ step bindings ⊆ declared deps │ │ Baton.DAG.validate — no cycles, dups, unknown deps │ └───────────────────────────────────────────────────────────┘ │ :ok ▼ ┌───────────────────────────────────────────────────────────┐ │ 2. LOAD CONTEXT │ │ an explicit opts[:context] wins outright │ │ else flow_context_provider().load(key, input, …) │ │ no provider → %{} │ │ provider but no key → :missing_context_provider_key │ └───────────────────────────────────────────────────────────┘ │ {:ok, context} ▼ ┌───────────────────────────────────────────────────────────┐ │ 3. EXPAND │ │ environment = %{input, context, steps: %{}, run: %{}}│ │ plain node → one expansion │ │ fan-out node → one expansion per collection item, │ │ id = "_" │ │ $steps fan-out → one expander, expands at run time │ │ logical deps remapped to expanded ids │ │ duplicate expanded ids rejected │ └───────────────────────────────────────────────────────────┘ │ {:ok, expansions} ▼ ┌───────────────────────────────────────────────────────────┐ │ 4. BUILD Baton.new + Baton.add │ │ one job per expansion, each stamped with a │ │ self-contained arg bundle │ │ llm → Workers.LLM, action → Workers.Action │ └───────────────────────────────────────────────────────────┘ │ ▼ ┌───────────────────────────────────────────────────────────┐ │ 5. SNAPSHOT Baton.put_flow_snapshot │ │ logical_definition + realized compiled_graph │ │ + definition_ref, format, input, context │ └───────────────────────────────────────────────────────────┘ │ ▼ %Baton.Flow.Compiled{workflow: Baton.t(), …} ``` Compilation produces a `Baton.Flow.Compiled` struct. Its `:workflow` field is an ordinary `Baton.t()` — insert it exactly like a hand-built one: ```elixir {:ok, definition} = Baton.Flow.Definition.load(attrs) {:ok, compiled} = Baton.Flow.Compiler.compile(definition, input: %{"document_id" => 42}, definition_ref: "version:7" ) {:ok, jobs} = Baton.insert(compiled.workflow) ``` ### Fan-out expansion A node carrying a `fan_out` spec expands into one node per collection item. The `gate` controls how the expanded nodes are scheduled relative to each other: ``` search ──▶ analyze ──▶ report gate: "parallel" analyze_1 ┐ for a 3-item collection becomes: analyze_2 ┼──▶ report analyze_3 ┘ gate: "sequential" analyze_1 ─▶ analyze_2 ─▶ analyze_3 ─▶ report ``` `parallel` gives every expanded node the step's declared deps, so they all run concurrently. `sequential` chains them: the first carries the declared deps and each later node additionally depends on the previous one. Choose `sequential` when the expanded nodes share a cacheable prompt prefix (the first call primes it and the rest read it), or to smooth a rate limit. The chaining edge is an *ordering* edge, not a dependency — it carries no data, and a predecessor that dies doesn't poison the expansions behind it. Note that `sequential` is incompatible with `"transport" => "batch"` (see the batch mode guide) and the validator rejects the pair. Note that downstream nodes depend on **every** expanded node — `report` above waits for all three analyses. A fan-out is also capped. `max_items` (default 200) refuses an expansion larger than that rather than inserting the jobs — a guard against a producer that degenerates, which is the failure dynamic fan-out exists to avoid, arriving from the other direction. ### Fanning out over a computed result The collection above came from `$input`/`$context`, so its size was known before any job ran. When the list is itself produced by an upstream node — claims an LLM just extracted — root the collection at `$steps.` instead: ```elixir %NodeSpec{ id: "assess", type: "llm", deps: ["extract"], fan_out: %FanOutSpec{ collection: "$steps.extract.data.claims", item_id: "$item.claim_id", max_items: 200 } } ``` Everything downstream is unchanged: one job per item carrying `$item`, and a reader that declares a dep on `assess` reads `$steps.assess` as the ordered list of results. What changes is *when* the expansion happens. The node compiles to a single **expander** step holding the logical id, which creates the rest when it runs and then waits for them (`Baton.Expansion`). Two consequences are worth knowing: - The expander adopts its own children as dependencies, so it sits in `scheduled` for the whole expansion and the workflow stays open until every child settles. - If any child ends non-completed the expander **discards**, mirroring what a reader would have seen from a static fan-out whose branch exhausted its retries. `ignore_discarded: true` on the reader is what lets it proceed with the partial list, exactly as it is for a static fan-out. Two rules the validator enforces. The collection must name a step the node declared a dep on — otherwise the expansion would read a step that has not run — and that step must not itself have fanned out, since reading through an expansion yields a list of lists whose meaning is ambiguous. > #### `item_id` must be an identifier, not a label {: .warning} > > Point `item_id` at a short stable id the producing node mints — an index, a > claim number, a slug. It becomes the expansion's step name, and step names > are `varchar(255)`. > > This is the one place where a dynamic fan-out is genuinely less forgiving > than a static one, and it is worth being deliberate about. A static > collection comes from your own input, so its suffixes are yours; a dynamic > one comes from an upstream result, which for an `llm` node means **model > output**. `"item_id": "$item.term"` over a list of claim terms reads > naturally and works right up until a model returns a whole limitation — > *"a set of security credentials that are each associated with corresponding > known computing systems"* is 97 characters, and verbose claim language goes > further. Ask the producing node for `claim_id` alongside `term` and fan out > over the id. > > An overflow is refused rather than truncated (truncating would silently > collide two items sharing a prefix): the expander discards with > `{:fan_out_item_id_too_long, node_id, length, limit}` on its first attempt, > because the collection is already computed and asking again cannot make the > suffix shorter. ### Fan-in `report` declared a dep on the logical `analyze`, but the three expansions stored their results under `analyze_1`, `analyze_2`, `analyze_3`. Reading `$steps.analyze` closes that gap: a fanned-out dependency resolves to its results as a **list, in expansion order**. ```elixir # in report's config "bindings" => %{"findings" => "$steps.analyze.data.finding"} #=> ["…claim 1…", "…claim 2…", "…claim 3…"] ``` A path segment applied to a list maps over its elements, so `.data.finding` plucks that path from each expansion instead of failing. The mapping is strict: if one element is missing the segment the whole expression is `{:binding_not_found, …}`, rather than a list quietly shorter than the fan-out. An expansion that stored no result at all is omitted. The reader always names the **logical** node. Expanded ids exist only at run time, so there is nothing to author against, and the validator rejects `$steps.analyze_1` for the same reason it rejects any undeclared dep — it would read part of an expansion while gating on all of it. Nothing changes for a dependency that did not fan out: `$steps.search` is that step's single stored result, as always. `Baton.Flow.Compiler` stamps each job with the expanded names of its fanned deps (`flow_fan_in`), which is what lets `Baton.Flow.Runtime` group them without reading the run snapshot. A dep that fanned out *dynamically* has no such stamp — its expansions did not exist when the reader was compiled — so its grouping is read from `workflow_nodes` instead, keyed by `fan_out_of` and ordered by `item_index`. The reader cannot tell the difference: `$steps.assess` is the ordered list either way, and a node may depend on one of each. ### Fanning out over models Because `model` is a binding, a fan-out over a list of models is just the ordinary pattern — the collection happens to be models rather than claims: ```elixir %NodeSpec{ id: "assess", type: "llm", config: %{"model" => "$item.id"}, fan_out: %FanOutSpec{collection: "$input.models", item_id: "$item.slug"} } ``` Compiled against `input: %{"models" => [%{"id" => "opus-5", "slug" => "opus"}, …]}` this yields `assess_opus`, `assess_gpt`, … each running the same prompt on its own model. A downstream node reads all of them with `$steps.assess`. ## 3. Compile time vs. run time This is the most important property of the design, and the pipeline diagram above only shows half of it. ``` COMPILE TIME (Compiler) RUN TIME (each Oban job) ─────────────────────── ──────────────────────── resolves: resolves: $input.… $steps.… upstream results $context.… $item, $run fan-out collection → N nodes prompts, model call, action the graph shape — ids, deps, edges ``` The split falls where it does for one reason: **a collection must be known in order to fix the graph.** A fan-out over `$input` or `$context` can therefore be expanded by the compiler, because both are available before any job runs. Per-item results do not exist yet, so anything reading `$steps.*` is deferred to run time. A fan-out over `$steps.*` is what happens when that deferral applies to the graph shape itself. The compiler cannot fix it, so it emits one expander step that fixes it later — the graph gains its remaining nodes mid-run, from a job running in the row above. The snapshot still records what was compiled; the expansion is recoverable from `workflow_nodes` (`fan_out_of`, `item_index`), which is also where a reader's fan-in list comes from. At run time each job reconstitutes its own environment from the args the compiler stamped on it: | Environment root | Source at run time | | --- | --- | | `input`, `context` | `job.args` (`flow_input`, `flow_context`) | | `steps` | `Baton.Results.get_all_results/1`, plus fanned deps grouped under their logical id (`flow_fan_in`) | | `run` | `job.meta["workflow_id"]`, `flow_definition_ref` | | `item` | `job.args["flow_item"]` | Because every job carries its own input and context, nodes stay independent — no shared mutable state, and a retry rebuilds the same environment. ## 4. Host seams The compiler never names a host module. Instead the host registers three implementations, looked up through `Baton.Config`: | Config key | Host contract | Resolved | Used by | | --- | --- | --- | --- | | `flow_context_provider` | `c:Baton.Flow.ContextProvider.load/3` | compile time | the compiler, to hydrate domain context | | `flow_prompt_resolver` | `c:Baton.Flow.PromptResolver.resolve/3` | run time | `Baton.Flow.Workers.LLM`, to render prompts | | `flow_registry` | `c:Baton.Flow.Registry.resolve_action/1`, `c:Baton.Flow.Registry.resolve_llm_adapter/1` | run time | both workers, to map string keys to modules | A node's `type`, `action`, and `adapter` strings are the indirection. This is what keeps definitions portable: the same JSON runs against any host that registers the keys it references. ```elixir config :baton, flow_context_provider: MyApp.Flows.ContextProvider, flow_prompt_resolver: MyApp.Flows.PromptResolver, flow_registry: MyApp.Flows.Registry ``` Actions are deliberately **allow-listed** (`Baton.Flow.Action`) rather than resolved by module name, so a stored definition can never name its way to an arbitrary function. ## 5. Properties and constraints Worth knowing before you build on this. **Reads are a subset of deps.** A node may only bind to steps it declared a dependency on; the validator rejects anything else. Dataflow and dependency edges are therefore the same graph, and no binding can bypass dependency gating. **Definitions are inert and versioned.** The `format` field versions the wire shape, and the flow snapshot pairs the logical definition with the `compiled_graph` it actually produced — enough to audit or replay a run. **Deterministic errors discard rather than retry.** `Workers.LLM` treats bad prompt variables and bad bindings as properties of the definition, not transient faults, and discards them immediately instead of consuming the job's retry budget. Transport and provider failures keep their normal retry and snooze behaviour. **Node types are a closed set.** `llm` and `action` are fixed in both the validator and the compiler's worker lookup. Adding a kind is a library change, not a host extension. **Config is the host's, except for `transport`.** Baton neither validates nor interprets what's in a node's `config` — prompts, bindings, schemas, and adapter keys are all resolved by host seams. The one exception is `transport` on an `llm` node, which the library reads to choose between a live call and the batch engine, and therefore also validates. `compile/2` can also default it per run (`transport: "batch"`), so one definition serves both an interactive and a bulk path — see the batch mode guide. **A dynamic fan-out cannot read another fan-out.** A `$steps.`-rooted collection must name a declared dep that did not itself fan out. Reading through an expansion resolves to a list per expansion, and whether that means "flatten" or "an expansion per expansion" is left undecided rather than guessed. **Nothing else grows the graph at run time.** Dynamic fan-out is the single exception to the rule that a workflow's shape is fixed at insert, and it is deliberately narrow: one node, expanding once, into siblings of itself.