Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

[0.1.3] - 2026-09-16

Fixed

• Acknowledgement advances only across a gapless prefix of committed events. Scriba acknowledged each batch as it committed. Acks are prefix acks — acking event 7 acknowledges everything up to 7 — and batches do not commit in source order when :parallelism exceeds 1. A handler still working on event 5, or sleeping in the retry loop, did not stop events 6 and 7 from committing and acking, which moved the subscription's checkpoint past event 5. A crash in that window lost event 5 outright: the store believed it had been delivered, the per-stream cursor never advanced, and nothing redelivered it. No dead letter, no cursor anomaly, no log line, and Scriba.info/2 still reporting :running.

  The producer now tracks dispatched events in delivery order and acknowledges only the longest run of committed ones with no gap. An uncommitted event holds the watermark back however many later events have committed; a crash then replays from below it, and pipeline-side dedup drops whatever had already been applied.

  0.1.2 users should upgrade. That release forwards :buffer_size and its documentation recommends raising it — which is exactly what opens this window. At the adapter default of one in-flight event, no later event can overtake a straggler, so the window is nearly unreachable; with buffer_size: 500 it is wide.

  Reproduced against a real EventStore in bench/test/ack_loss_test.exs, which fails on 0.1.2 and passes here.

  Also faster: one acknowledgement per drain rather than one per event removes a GenServer.call per event from the commit path. The benchmark went from 2,448 to 4,761 events/sec at buffer_size: 500.

[0.1.2] - 2026-09-16

Added

• :buffer_size, :concurrency_limit and :partition_by are forwarded to the event store subscription. Scriba.Source.Commanded previously subscribed with no options at all, so the adapter's defaults always applied. EventStore's default is one in-flight event per subscriber, and that — not :parallelism — is what bounds catch-up: Scriba acknowledges after the batch commits, so a batcher holding a single event waits out its full :batch_timeout before acking and releasing the next.

  Measured against a real EventStore, 5,000 events over 100 streams: 9.1 events/sec at the adapter default, 2,448 events/sec with buffer_size: 500.

  Scriba sets no default of its own — the adapter's still applies unless configured, so this changes nothing for an existing projection until it opts in. Raising the buffer trades memory and redelivered-work-after-a-crash for throughput.

  source: {Scriba.Source.Commanded,
           application: MyApp.CommandedApp,
           buffer_size: 500}

Fixed

• Acknowledgement now comes from the process that holds the subscription. Against commanded_eventstore_adapter — or any adapter that identifies the acking subscriber by self() — every acknowledgement Scriba issued was silently discarded. The subscription never advanced, and the projection stalled permanently once the event store's in-flight buffer filled. Measured against a real EventStore: one event projected, then nothing, with status still :running, no error, no log line.

  ack/3 is a Broadway acknowledger, so it runs in a batch-processor process rather than the producer that subscribed. It called ack_event/3 from there. Commanded.EventStore.Adapters.InMemory takes the subscription as an argument and ignores the caller, so acks worked; EventStore resolves the subscriber from self() and its subscription FSM drops acks from any pid it does not recognise. The engine was correct only against the adapter it was tested with.

  The batch processor now hands its acknowledgements to the producer, which issues them in delivery order. Both adapters see an ack from a pid they recognise. No public API, configuration, schema or telemetry change.

  Anyone running Scriba against a persistent event store should upgrade: on 0.1.0 and 0.1.1 the projection stops after the first batch and does not report that it has.

[0.1.1] - 2026-07-30

Fixed

• A halted projection is now queryable, not just observable at the instant it halts. Scriba.info/2 reported :running for a projection that had hit a structural commit failure and would never move again — the halt was announced once via [:scriba, :projection, :halted] and a log line, and was invisible from then on. An operator who was not subscribed to telemetry at that moment had a stopped projection and no way to see it, which is the same silent-stall shape the halt path exists to replace.

  The Pipeline now reports structural failures to the Coordinator, which carries a terminal :halted state. Scriba.info/2 exposes it as :status along with a new :halt_reason field naming the cause (typically a Postgrex.Error whose SQLSTATE identifies it). Polling status is enough to detect a stopped projection.

  pause/1 and resume/1 are rejected from :halted; stop/1 is the way out once the underlying schema or permission is fixed. Additive — no existing field or return value changed.

0.1.0 - 2026-06-10

Initial release. Projection engine for Elixir event-sourced systems that pairs with Commanded, replacing the role commanded_ecto_projections filled before it stopped being actively maintained.

Added

Public API

• use Scriba.Projection macro. Five-line declaration generates the full projection module:

  use Scriba.Projection,
    name: "orders",
    source: {Scriba.Source.Commanded, application: MyApp.CommandedApp},
    target: {Scriba.Target.Ecto, repo: MyApp.Repo},
    parallelism: 16

  Compile-time validation rejects unknown options, validates :partition_by is :stream_id (the only value supported in v0.1), and warns when :name matches the legacy _v<integer>$ suffix pattern from commanded_ecto_projections.
• Scriba.start_projection/1 and /2. Adds the projection to Scriba.Projections.Supervisor (a DynamicSupervisor) at runtime; /2 accepts overrides for non-identity options (:parallelism, :retry, etc.) and rejects :name / :version overrides since identity is compile-time.
• Scriba.pause/1, resume/1, stop/1, info/1, list/0. Each accepts either a projection module (reads identity from the generated __scriba_config__/0) or a string name (defaults to version 1). pause/2, resume/2, stop/2, info/2 take explicit (name, version) for operator workflows.

Engine internals

• Per-stream cursor tracking in scriba_positions. One row per (projection_name, projection_version, stream_id) — cross-partition commit ordering can no longer cause silent drops because each stream has its own cursor.
• Atomic position + read-model commit. Every successful batch commits the user's read-model writes AND every touched stream's cursor advance in one Ecto.Multi transaction. No observable state where the read model advanced but the cursor didn't, or vice versa.
• Consistent-hash partitioning by stream_id via Scriba.Partitioner.partition/2. Events on the same stream deterministically route to the same Broadway processor; events across streams parallelize up to :parallelism.
• Source-side dedup at Pipeline.handle_message/3. Events whose position is at or below the committed cursor for their stream return :skip without invoking the handler — protects against source redelivery after Pipeline restart.
• Shared ETS cache for hot-path cursor reads (Scriba.Position.Cache). One named table for the whole BEAM, keyed by {name, version, stream_id} tuples. Replaces the earlier per-projection named-table design that allocated one BEAM atom per projection.

Source adapters

• Scriba.Source.Commanded. Subscribes to a Commanded.Application via Commanded.EventStore.subscribe_to/5; converts Commanded.EventStore.RecordedEvent structs into Broadway messages with %Scriba.Event{} data. Acknowledges back to Commanded after the Multi commits. :commanded is an optional: true dep — Scriba compiles and runs without it.
• Scriba.Source.Commanded.pause/1 and resume/1 callbacks. Pause flips an internal flag; handle_demand/2 accumulates demand without dispatching until resume. The EventStore subscription keeps pushing into the source's queue during pause — bounded memory growth documented in the module's moduledoc.

Target adapters

• Scriba.Target.Ecto. Builds an Ecto.Multi from per-event handler results plus per-stream cursor advances plus per-event dead-letter inserts; runs everything in a single Repo.transaction/1. Handler returns become Multi steps:
  • :skip → no step (cursor still advances per-stream).
  • {:insert, struct} → Ecto.Multi.insert/3.
  • {:update, schema, filter, [set: changes]} → Ecto.Multi.update_all/4.
  • {:delete, schema, filter} → Ecto.Multi.delete_all/3.
  • {:multi, %Ecto.Multi{}} → merged via Ecto.Multi.merge/2.
• Scriba.Target.Test. In-memory Agent-backed target for property tests and downstream user testing. Records commits, per-stream positions, and dead-letter entries.

Operational features

• Telemetry events (full surface in Scriba.Telemetry's moduledoc):

  • [:scriba, :projection, :event, :start | :stop | :exception] — via :telemetry.span/3 around each handler invocation. Fires per retry attempt.

  • [:scriba, :projection, :batch, :stop] — manual emit after Multi commit succeeds. Measurements %{duration, batch_size}.
  • [:scriba, :projection, :dead_letter] — once per dead-lettered event after the Multi commits. Metadata %{projection, position, stream_id, event_type, error_kind}.

  • [:scriba, :projection, :started | :paused | :resumed] — Coordinator lifecycle events. :started fires once per Coordinator-process lifetime (Pipeline DOWN → re-up does NOT re-fire).

  • [:scriba, :projection, :cache_initialized] — ETS preload from Postgres at Coordinator start.
  • [:scriba, :source, :batch, :failed] — a batch failed to commit and nothing in it was acknowledged. Measurements %{count}, metadata %{subscription, reason}.
  • [:scriba, :projection, :halted] — a structural commit failure stopped the projection. Metadata %{projection, reason, failure} where failure is the SQLSTATE label. The event to page on.
• Commit failures are classified by SQLSTATE, not guessed (Scriba.Failure). :integrity (classes 22/23, Ecto.ConstraintError, invalid changesets) is deterministic and event-specific — the batch is re-applied one transaction per event so the offender dead-letters with error_kind "commit:<SQLSTATE>" and the rest commit. :transient (classes 08/53, 40001, 40P01, 57014, DBConnection errors) replays. :structural (class 42 and anything unrecognised) halts the projection and emits [:scriba, :projection, :halted], because replaying loops forever and dead-lettering would destroy a batch over a fixable deploy-ordering mistake. In the per-event pass a stream stops at its first unresolved event rather than skipping past it, preserving per-stream ordering and preventing a cursor gap.
• Batch commit failures replay; they never partially acknowledge. When a target's transaction does not commit, nothing in the batch is acknowledged — including the messages that succeeded, because event store acks are prefix-acks and cannot express a gap. The source stops its producer, which rewinds the subscription to its last durable checkpoint; the batch is redelivered and source-side dedup filters whatever did commit. See Scriba.BatchCommitError.
• Malformed handler returns dead-letter instead of wedging the projection. A return outside §4.2's six shapes is routed to scriba_dead_letters with error_kind "invalid_return" and the cursor advances. Targets declare their own vocabulary through the optional Scriba.Target.valid_result?/1 callback, so this does not hard-code the Ecto target's shapes into the engine.
• Duplicate Ecto.Multi operation names within a batch are detected before assembly. Scriba merges a whole batch into one Multi, so two events naming an operation identically would collide inside Repo.transaction/1 — deterministically, on every redelivery. The first claim now wins and later claimants dead-letter individually with error_kind "multi_key_collision". Scriba's own step keys ({:scriba_event, _}, {:scriba_position, _}, {:scriba_dead_letter, _}) are reserved.
• Cursor monotonicity is enforced in the schema. The scriba_positions upsert is GREATEST(existing, incoming), so a cursor cannot move backwards regardless of what the Pipeline computes.
• Restart intensity is chosen, not defaulted. Scriba.Projection.Supervisor runs at 30 restarts per 60 seconds. A commit failure now kills the producer on purpose, so the OTP default of 3-in-5 would exhaust during any real outage and propagate toward the host application; 30-in-60 absorbs a multi-minute outage while still bounding a genuine crash loop.
• Resubscribe tolerates the reap race. After a producer dies to force a replay, the event store may not yet have processed its DOWN, so resubscribing can transiently see :subscription_already_exists. That is now retried with backoff (~1.5s total) and, if it persists, reported as a name conflict rather than as a migration problem.
• Dead-letter routing. Handler {:error, _} returns and raised exceptions route to scriba_dead_letters after retry exhaustion, with the cursor advancing past the failed event (skip-and-continue per architecture §9.2). The dead-letter insert and the cursor advance commit atomically in the same Multi as the batch's successful writes.
• Retry policy with exponential backoff (default 3 attempts at 100ms / 1s / 10s) configurable per projection via retry: [max_attempts: N, backoff: [...]] in use Scriba.Projection. retry: false opts out for immediate dead-letter. Implementation is an in-handler Process.sleep loop — see SCRIBA_ARCHITECTURE.md §9 for why this is the right primitive rather than Process.send_after/3 against Broadway's processor model.
• Real pause / resume with held demand. The Coordinator's :running → :paused transition signals the source's GenStage producer to stop yielding; the Pipeline tree stays alive, in-flight events finish their commit lifecycle, accumulated demand drains on resume. Replaces the earlier terminate_child/restart_child implementation that was misleadingly named.
• Scriba attaches no telemetry handlers of its own. It emits events and gets out of the way; users attach their own in their app's start/2. Default handlers, if any, are a v0.2 concern and will arrive with the behaviour that justifies them.

Migrations

• Scriba.Migrations.up/0 and down/0. Users call these from their own Ecto migration:

  defmodule MyApp.Repo.Migrations.AddScribaTables do
    use Ecto.Migration
    def up,   do: Scriba.Migrations.up()
    def down, do: Scriba.Migrations.down()
  end

  Creates scriba_positions (per-stream cursors) and scriba_dead_letters.

Documentation

• MIGRATION.md — migrating from commanded_ecto_projections. project/2,3 → handle/2, the Ecto.Multi differences (one transaction per batch, not per event), the meta map mapping, callbacks with no equivalent (after_update/3, schema_prefix/1, consistency: :strong), and cursor carry-over: both libraries track Commanded's global event_number, so last_seen_event_number passed as :start_from cuts over in place with no read-model rebuild. :start_from exclusivity verified against commanded_eventstore_adapter + eventstore.
• Legacy use Scriba.Projection options raise with targeted guidance rather than a generic unknown-option error — :consistency most of all, since it is the only migration gap that is invisible at runtime.
• SCRIBA_ARCHITECTURE.md — the architectural contract. Non-negotiable invariants, supervision tree, Coordinator state machine, position-tracking storage shape, error handling, property tests, dependencies, file layout, behaviour signatures.
• Module-level moduledocs on every public module documenting the contract from the user's perspective, including sharp edges (Scriba.Source.Commanded's pause-memory caveat, Scriba.Target.Ecto's {:multi, _} escape hatch for SQL-level increments, etc.).

Example application

• examples/bank/ — self-contained Mix project demonstrating the five-line API end-to-end. Real Commanded (Commanded.EventStore.Adapters.InMemory for fast iteration), real Ecto, real read model. Three accounts, fifty random deposits/withdrawals, the projection converges and the demo prints the final balances. mix bank.setup + mix bank.demo runs in under 30 seconds against a local Postgres.

Fixed

• Scriba.Source.Commanded.to_message/2 read the wrong event-struct field. It read stream_uuid from Commanded's RecordedEvent, but the field in Commanded 1.4 is stream_id. Any event delivered from a real Commanded subscription would have raised KeyError on the first message. The bug was present in the initial Source.Commanded implementation and went undetected because that path had no end-to-end coverage until the Commanded integration tests were added during pre-release verification (commit 2778ae9). It never shipped in a tagged release — fixed before v0.1.0. Anyone evaluating Scriba against Commanded from a pre-release snapshot who hits a stream_uuid KeyError should move to v0.1.0.

Property tests

The three architectural invariants from SCRIBA_ARCHITECTURE.md §10 are mandatory for v0.1 and covered by property tests:

• P1 — Per-stream ordering. The sequence of events delivered to handle/2 for any stream is a prefix of the source order for that stream.
• P2 — Position monotonicity. Committed positions in Postgres are monotonically non-decreasing across all observed states, including across simulated worker crashes (200 iterations against real Postgres).
• P3 — Effectively-once under crash. For each event, the handler's effect appears in the target exactly once after recovery from arbitrary crash schedules (100 iterations against real Postgres for the cursor-resume variant; the integration-test side of the property is in test/scriba/projection/pipeline_test.exs).

Dependencies

• :broadway ~> 1.1
• :ecto_sql ~> 3.11
• :postgrex ~> 0.17
• :telemetry ~> 1.2
• :jason ~> 1.4
• :commanded ~> 1.4 (optional)

Dev / test:

• :stream_data, :ex_doc, :credo, :dialyxir.

Architectural decisions worth knowing

These are documented at length in SCRIBA_ARCHITECTURE.md but worth calling out for users upgrading from commanded_ecto_projections or similar:

• name and version are separate fields. Do not encode the version in the name ("orders_v1" is wrong; the macro warns at compile time). The architecture supports running multiple versions side-by-side during a cutover via two modules with different version: integers.
• Dead-lettered events advance the cursor. Blocking the projection on a bad event is the #2 complaint on ElixirForum after lag visibility (per architecture §9.2). Scriba defaults to skip-and-continue with dead-letter visibility via telemetry; users who need block-on-failure semantics build that on top.
• No idempotency on pause/resume. pause on :paused and resume on :running return {:error, {:invalid_state, _}}, not :ok. Silent idempotency hides bugs. Callers wanting "make sure this is paused" semantics check Scriba.info/1 first or pattern-match the matching-state error case as success.
• Multi transaction failures do not retry through the per-event retry policy. Retry wraps the handler call, not the Multi commit. Database-level failure leaves the batch unacked; the source re-delivers when the projection's Pipeline next runs.