# Scriba A projection engine for Elixir event-sourced systems. Pairs with Commanded for the event-sourcing side; replaces the role [`commanded_ecto_projections`](https://github.com/commanded/commanded-ecto-projections) used to play before it stopped being actively maintained. ```elixir defmodule MyApp.Projections.Orders do use Scriba.Projection, name: "orders", source: {Scriba.Source.Commanded, application: MyApp.CommandedApp}, target: {Scriba.Target.Ecto, repo: MyApp.Repo}, parallelism: 16 def handle(%OrderPlaced{} = event, _meta) do {:insert, %OrderReadModel{id: event.order_id, status: "pending"}} end def handle(%OrderShipped{order_id: id}, _meta) do {:update, OrderReadModel, [id: id], set: [status: "shipped"]} end def handle(_, _), do: :skip end {:ok, _} = Scriba.start_projection(MyApp.Projections.Orders) ``` That's the API. The rest is operational scaffolding you get for free. **Already running `commanded_ecto_projections`?** Read [`MIGRATION.md`](MIGRATION.md). Short version: your existing cursor carries over — both libraries track Commanded's global `event_number`, so you hand your `last_seen_event_number` to Scriba as `:start_from` and cut over in place. No read-model rebuild, no maintenance window. --- ## Status **v0.1.** The architectural contract is frozen ([`SCRIBA_ARCHITECTURE.md`](SCRIBA_ARCHITECTURE.md)) and the operational primitives — telemetry, dead-letter routing, retry policy, real pause/resume — are in place. Property tests cover per-stream ordering, position monotonicity, and effectively-once delivery under crash. What's deliberately out of scope for v0.1: - Lag and throughput metrics beyond raw telemetry events (v0.2 + LiveView dashboard). - Online rebuild / shadow targets / atomic swap (v0.3). - Sources other than Commanded; targets other than Ecto/Postgres (v0.4). - Multi-target fan-out (v0.4). See [`SCRIBA_ARCHITECTURE.md`](SCRIBA_ARCHITECTURE.md) §2 for the full in-scope / out-of-scope split. --- ## Why this exists You're running Commanded. You need read models. The library you'd have reached for — [`commanded_ecto_projections`](https://github.com/commanded/commanded-ecto-projections) — has been unmaintained for years and has known sharp edges around crash recovery, per-stream ordering across parallelism, and lag visibility. Scriba is an opinionated rewrite of that role with three principles: 1. **Correctness over throughput.** Position update and read-model write commit atomically inside a single `Ecto.Multi`. There is no configuration that lets you turn this off, because you should not want to. 2. **Per-stream ordering is preserved.** All events for a given `stream_id` route to the same processor via consistent hash. Within a processor, events are serial. Across streams, events are parallel. 3. **Sharp edges are documented, not hidden.** Dead-lettered events advance the cursor (skip-and-continue, not block-the-projection). The handler return contract is six tagged tuples, not a DSL. Lag is a telemetry-consumer concern, not an engine feature. Migrating an existing projector is a mechanical rewrite — `project %Event{}, fn multi -> ... end` becomes `def handle(%Event{}, meta)` returning a tagged tuple. [`MIGRATION.md`](MIGRATION.md) covers the rewrite, the `Ecto.Multi` differences, the callbacks with no equivalent (`after_update/3`, `schema_prefix/1`, `consistency: :strong`), and cursor carry-over. --- ## Installation ```elixir defp deps do [ {:scriba, "~> 0.1"}, # Optional — needed only if you use Scriba.Source.Commanded, # which is the only source shipped in v0.1. {:commanded, "~> 1.4"} ] end ``` `:commanded` is Scriba's one optional dependency: nothing in the engine references it statically, so Scriba compiles without it, and `Scriba.Source.Commanded.start_link/1` raises with instructions if you configure the Commanded source without adding it. Everything else arrives transitively and is **not** optional — `:ecto_sql` and `:postgrex` back the position cursor and dead-letter tables (`Scriba.Position`, `Scriba.DeadLetter`, `Scriba.Migrations`), not merely `Scriba.Target.Ecto`; `:broadway` is the pipeline runtime; `:telemetry` and `:jason` are used throughout. You do not list them yourself, but they will be in your dependency tree. ### Scriba is a Broadway topology Worth knowing before you read the failure-modes section below, which is written in Broadway's vocabulary: each projection is a [Broadway](https://hexdocs.pm/broadway) pipeline. The source is a Broadway producer, `:parallelism` sets the processor concurrency, `:batch_size` / `:batch_timeout` configure the batcher, and event acknowledgement runs through a Broadway acknowledger that Scriba implements against your event store. You never write Broadway code — but when this README says "the batch is marked failed", that is `Broadway.Message.failed/2`, and Broadway is where the retry-on-redelivery behaviour comes from. Add a migration to your repo for Scriba's tables: ```elixir defmodule MyApp.Repo.Migrations.AddScribaTables do use Ecto.Migration def up, do: Scriba.Migrations.up() def down, do: Scriba.Migrations.down() end ``` This creates two tables in your read-model database: - `scriba_positions` — per-stream cursor (one row per `{projection, version, stream_id}`). - `scriba_dead_letters` — failed events for inspection / manual replay. Start projections during your application boot. The idiomatic pattern is a small `Task` in your supervision tree that calls `Scriba.start_projection/1` once Ecto and Commanded are up: ```elixir # Bank.Application or equivalent children = [ MyApp.Repo, MyApp.CommandedApp, MyApp.Projections.Starter # Task that calls Scriba.start_projection ] ``` See [`examples/bank/lib/bank/projections/starter.ex`](https://github.com/thatsme/scriba/blob/main/examples/bank/lib/bank/projections/starter.ex) for the working pattern. --- ## Core concepts ### `name` and `version` A projection's identity is `(name, version)`. `name` is the stable logical label ("orders"). `version` is an integer (default `1`) you bump when you want to run a *new* projection side-by-side with the old one during a cutover. ```elixir # orders v1 — the current production projection defmodule MyApp.Projections.OrdersV1 do use Scriba.Projection, name: "orders", version: 1, ... end # orders v2 — running alongside, populating a new read model defmodule MyApp.Projections.OrdersV2 do use Scriba.Projection, name: "orders", version: 2, ... end ``` **Do not** encode the version in the name (`name: "orders_v2"`). The macro warns at compile time when it sees that pattern, because `commanded_ecto_projections` historically used it and it makes side-by-side versioning awkward. See architecture §5. ### Handler return shapes Your `handle(event, meta)` clauses must return one of these six values: | Return | Effect | |---|---| | `:skip` | Event acknowledged, no read-model write. Cursor still advances. | | `{:insert, schema_struct}` | `Ecto.Multi.insert/3` | | `{:update, schema_module, filter_keyword, [set: keyword]}` | `Ecto.Multi.update_all/4` filtered by the keyword | | `{:delete, schema_module, filter_keyword}` | `Ecto.Multi.delete_all/3` | | `{:multi, %Ecto.Multi{}}` | Merged into the batch's Multi — escape hatch for `:inc`, complex queries, etc. | | `{:error, reason}` | Routes to dead-letter (after retries, if enabled). | Raising an exception is also valid; it's converted to a dead-letter entry whose `error_kind` is the exception's module name. Per architecture §4.2. ### The `meta` map The second argument to `handle/2`: ```elixir %{ id: "event-uuid", stream_id: "aggregate-uuid", type: "OrderPlaced", position: 42, metadata: %{correlation_id: "..."}, occurred_at: ~U[2026-05-14 12:00:00Z] } ``` Use `:id` for idempotency keys (globally unique). Use `:position` for ordering within Scriba's internal accounting; don't use it as a stable identifier across event-store rebuilds. ### Per-stream ordering Events with the same `stream_id` route to the same processor via consistent hashing — different events on the same stream are *never* processed concurrently. Events across streams *are* parallel, bounded by `:parallelism`. This is the invariant that makes "balance += amount" projections correct without explicit locking. ### Atomic position commit Every successful batch commits the user's read-model writes AND the per-stream cursor advances in **one** `Ecto.Multi` transaction. There is no observable state where the read model advanced but the cursor didn't — or vice versa. This is the property that makes crash recovery work: on Coordinator restart, the source is told to resume from the durably committed cursor, and Scriba's source-side dedup skips any events the upstream re-delivers below that cursor. --- ## Operational features ### Telemetry Twelve events fire — from the Pipeline, the Coordinator, position-cache init, and the source. The full surface table is in `Scriba.Telemetry`'s moduledoc and in architecture §6.3. Highlights: ``` [:scriba, :projection, :event, :start | :stop | :exception] [:scriba, :projection, :event, :skipped] [:scriba, :projection, :batch, :stop] [:scriba, :projection, :dead_letter] [:scriba, :projection, :started | :paused | :resumed] [:scriba, :projection, :cache_initialized] [:scriba, :projection, :halted] [:scriba, :source, :batch, :failed] ``` `:event :stop` fires once per successful handler invocation — counting these gives you exact "events processed" without doing position-arithmetic across streams. `:dead_letter` fires once per dead-lettered event with `{name, version, position, stream_id, event_type, error_kind}` metadata for alerting. `[:scriba, :source, :batch, :failed]` means a batch did not commit, nothing in it was acknowledged, and the source is restarting to replay from its last durable checkpoint. Isolated occurrences are normal under transient database trouble; a sustained stream of them means no progress. `[:scriba, :projection, :halted]` is the page. The projection hit a structural failure — a column that doesn't exist, a missing privilege — and stopped on purpose, because replaying it would loop forever and dead-lettering it would destroy a batch over a fixable deploy-ordering mistake. Nothing is lost; nothing proceeds either. The metadata carries the SQLSTATE. Scriba attaches no handlers of its own — it emits and gets out of the way, so it never competes with your observability stack. You write your own with `:telemetry.attach_many/4` in your application's `start/2`. ### Dead-letter routing A failing event — handler returned `{:error, _}` or raised, AFTER retry exhaustion — gets a row inserted into `scriba_dead_letters` **atomically with the cursor advance**. The projection does not block on bad events. This is a deliberate sharp edge, documented at architecture §9.2: > "When an event is dead-lettered, the position advances past it. > The alternative — blocking the projection until the bad event is > resolved — is the #2 complaint on ElixirForum after lag visibility." If you want block-on-failure semantics for a specific projection, you build that on top: subscribe to `[:scriba, :projection, :dead_letter]` telemetry, page someone, manually replay from the dead-letter table once they've fixed the underlying issue. ### Retry policy Default: 3 attempts with exponential backoff (100ms, 1s, 10s) before dead-letter. Configurable per projection: ```elixir use Scriba.Projection, ... retry: [max_attempts: 5, backoff: [100, 500, 2000, 10_000]] ``` Or `retry: false` to opt out (one attempt, immediate dead-letter on failure). The retry loop is in-handler `Process.sleep` — see architecture §9.1 and §9 for why this is the right primitive rather than `Process.send_after` (Broadway's processor model). Each retry attempt re-invokes `:telemetry.span/3`, so per-attempt `:event :start` / `:event :stop` / `:event :exception` events fire. Operators counting `:event :start` per `event_id` can see retry activity without a dedicated `:retry` event. ### Pause / resume `Scriba.pause(MyApp.Projections.Orders)` signals the source to stop yielding new events. The Pipeline tree stays alive; in-flight events finish their commit lifecycle. `Scriba.resume(...)` reverses the signal. Both fire telemetry; both are honest about asynchrony (the source-pause signal is `send/2`, so by the time `pause/1` returns the signal is in the source's mailbox but the source's `handle_info` may not yet have run). `pause` on `:paused` and `resume` on `:running` return `{:error, {:invalid_state, _}}` — they are deliberately **not idempotent**. Silent idempotency hides bugs. If you want "make sure this is paused" semantics, check `Scriba.info/1` first or pattern-match the matching-state error case as success. --- ## Failure modes worth knowing ### Handler raises Re-raise is caught by Scriba, the exception is logged as a `:event :exception` telemetry event, and the event enters the retry loop. After retry exhaustion, the original exception's struct and stacktrace are recorded in `scriba_dead_letters` with `error_kind` equal to the exception module name (e.g. `"Elixir.ArgumentError"`). ### Multi transaction fails What happens depends on *why*, and Scriba reads that from the SQLSTATE rather than guessing (`Scriba.Failure`). Guessing from "did some events succeed?" is wrong in both directions: resource pressure fails non-uniformly, so a partial success looks deterministic when it isn't; and handler code deployed ahead of its migration fails uniformly, so it looks transient when it very much isn't. **Transient** (connection loss, deadlock, serialization failure, resource exhaustion, cancelled query). Nothing in the batch is acknowledged — including the events that succeeded, because event-store acks are prefix-acks and cannot express a gap. The source stops its producer, the subscription rewinds to its last durable checkpoint, and the batch is redelivered. Source-side dedup filters whatever did commit. **Integrity** (unique violation, NOT NULL, foreign key, check constraint, numeric overflow). Deterministic and specific to one event, so replaying it forever is pointless. The batch is re-applied one transaction per event: the offending event lands in `scriba_dead_letters` with `error_kind` `"commit:23505 (unique_violation)"` or similar, the rest commit, and the projection keeps moving. **Structural** (undefined column or table, insufficient privilege, and anything Scriba cannot classify). Neither response is safe — dead-lettering would destroy a batch over a fixable deploy-ordering mistake, replaying would loop forever — so the projection **halts** and says so via `[:scriba, :projection, :halted]` and a log line naming the SQLSTATE. Nothing is acknowledged and no cursor moves, so nothing is lost. It resumes when you fix the cause and restart. **Multi failures do not retry through the per-event retry policy.** The retry layer wraps the handler call, not the Multi commit. If your DB is intermittently failing, you want it to recover at the DB level — not for individual events to retry-then-dead-letter against a sick database. Per-stream ordering survives all three: in the per-event pass a stream stops at its first unresolved event rather than skipping past it. > **Verification status.** The replay path — the source refusing to > acknowledge, killing its producer, rewinding the subscription, backing off > and redelivering — is covered by **unit tests only**. It has never been > exercised against a real event store, and conservation across a database > outage (`events delivered == read-model rows + dead letters + skipped`) is > unverified. The reason is that Scriba's test double, `Scriba.Test.Source`, > discards failed messages instead of redelivering them, so the suite cannot > replay anything. The transient/integrity/structural classification above > *is* verified against real Postgres, including that a `docker stop` emits > `57P01` and is treated as transient. ### Source redelivers events the projection has already committed This happens after Pipeline restart — Commanded's subscription resumes from its acked position, which may be behind Scriba's durably committed position. Source-side dedup at the Pipeline's `handle_message/3` (architecture §3.4) catches these: events whose position is at or below the committed cursor for their stream are returned as `:skip` without invoking the handler. This is the property `P3 — effectively-once under crash` exists to verify. Its real-Postgres property test is `test/property_db/pd3_cursor_resume_test.exs` (PD3 — recovery resumes from the committed cursor, 100 iterations against real Postgres), with the integration-test side in `test/scriba/projection/pipeline_test.exs`. --- ## Example app [`examples/bank/`](https://github.com/thatsme/scriba/tree/main/examples/bank) is a self-contained Mix project that demonstrates the full path: real Commanded (`Commanded.EventStore.Adapters.InMemory` for fast iteration), real Ecto, real read model. The projection module itself is ~50 lines including comments. ```sh cd examples/bank mix deps.get mix bank.setup # creates the database, runs migrations mix bank.demo # opens 3 accounts, dispatches 50 random ops, prints balances ``` The demo's wait-for-completion uses a telemetry counter on `:event :stop` — exactly the pattern you'd reach for in your own test code. See [`examples/bank/lib/mix/tasks/bank.demo.ex`](https://github.com/thatsme/scriba/blob/main/examples/bank/lib/mix/tasks/bank.demo.ex). The example app is **not** included in the Hex package tarball — these links go to GitHub. Clone the repo to run it. --- ## Running Scriba's own tests | Command | What runs | Requires | |---|---|---| | `mix test.fast` | Non-property tests | — | | `mix test.property_db` | Real-Postgres property tests in `test/property_db/` | `SCRIBA_TEST_DB_*` env vars | | `mix test.all` | Everything ExUnit will run with the current environment | — for always-runnable parts; env vars for `property_db` | `mix test.all` is the canonical full-suite command. Status reports should name the command: "fast suite: 99/0" rather than "99 tests, 0 failures." Naming the command in reports avoids the ambiguity that bit us during the diagnostic phase. ### Real-Postgres property tests Property tests in `test/property_db/` (Phase B of the rebuild) need a live Postgres. Set all five of: | Variable | Example | |---|---| | `SCRIBA_TEST_DB_HOST` | `localhost` | | `SCRIBA_TEST_DB_PORT` | `5432` | | `SCRIBA_TEST_DB_NAME` | `scriba_test` | | `SCRIBA_TEST_DB_USER` | `postgres` | | `SCRIBA_TEST_DB_PASS` | `postgres` | Local convenience: copy `.env.local.example` to `.env.local` and fill in real values. `config/test.exs` auto-loads it; `.env.local` is gitignored. Shell environment variables override `.env.local`. **Policy is all-or-nothing-or-error:** all five set → tests run; all five unset → tests excluded with a startup message; any subset partially set → `config/test.exs` raises at config load. Partial configuration is treated as a misconfiguration, not a graceful degrade. The database must already exist; `mix test` does not create it. Migrations run in `test_helper.exs` against an existing connection. --- ## Documentation - [`MIGRATION.md`](MIGRATION.md) — migrating from `commanded_ecto_projections`: `project/2` → `handle/2`, `Ecto.Multi` differences, and how to carry your existing cursor across so you cut over in place instead of rebuilding read models. - [`SCRIBA_ARCHITECTURE.md`](SCRIBA_ARCHITECTURE.md) — the architectural contract. Read this before opening a PR that changes engine behavior. - [`examples/bank/README.md`](https://github.com/thatsme/scriba/blob/main/examples/bank/README.md) — example app walkthrough. - `Scriba.Telemetry` moduledoc — full v0.1 telemetry event surface. --- ## Versioning Scriba follows semver. The v0.1.0 API surface (the seven functions in `Scriba` plus the macro at `Scriba.Projection`) is frozen — no breaking changes within 0.1.x. New optional features may land in 0.1.x patch releases. `Scriba.Target` and `Scriba.Source` behaviours are not yet frozen — v0.4 will widen them when non-Commanded sources and non-Ecto targets land. Custom adapter authors should pin against a specific 0.1.x. --- ## License Apache-2.0. See [LICENSE](https://github.com/thatsme/scriba/blob/main/LICENSE). --- ## Contributing Issues and PRs welcome at https://github.com/thatsme/scriba. Architecture-affecting changes should reference the relevant section of `SCRIBA_ARCHITECTURE.md`; deliberate departures from the contract require documentation in the PR explaining why.