How Threadline works

Copy Markdown View Source

This is the crash-course guide. Read it when you want the mental model first and the API details second. For the exact seam contracts, keep using the integration contracts, operator surface, domain reference, and Phoenix SaaS getting-started guide.

The short version

Threadline is embedded audit infrastructure for Phoenix, Ecto, and PostgreSQL apps.

The memorable formula is:

DB truth + app intent + operator tooling

  • DB truth = trigger-captured AuditTransaction + AuditChange
  • app intent = semantic AuditAction records — normally via Threadline.Audit.transaction/3 with :action (implemented by Threadline.record_action/2 inside the helper)
  • operator tooling = timelines, actor windows, incident bundles, exports, and the optional /audit surface

Threadline is:

  • a library you embed into your app
  • a read-heavy investigation surface with a small write-side semantic helper
  • host-owned on auth, tenancy, and roles

Threadline is not:

  • event sourcing
  • a remote SaaS
  • an auth framework
  • a write-capable admin backend

The evidence plane stays just as narrow. Threadline may persist evidence about its own governance surfaces such as redaction posture, trigger coverage, retention runs, export delivery, and support-scope posture.

Canonical public non-goals for the evidence plane:

  • legal hold workflows
  • immutable-storage guarantees beyond the host runtime/storage contract
  • generic compliance packs
  • vendor-specific reporting suites
  • a Threadline-owned RBAC platform
  • a Threadline-owned tenancy DSL
  • approval workflows

The flow

The common loop is:

  1. A request enters the host app.
  2. Threadline.Plug attaches request-scoped audit context and the host decides the actor.
  3. Domain writes run inside Threadline.Audit.transaction/3 — the recommended audited write path — so capture and optional semantics share one database transaction.
  4. PostgreSQL triggers capture the physical row changes from that transaction.
  5. When you pass :action, the helper records semantic intent via Threadline.record_action/2 and links audit_transactions.action_id for correlation filters.
  6. Operators inspect the result through the query APIs, Mix tasks, or the mounted /audit surface.

Request headers can populate audit context at the edge, but queryable correlation requires an audit_actions row linked in the same transaction — use Audit.transaction/3 with :action when filters must match intent. Omit :action or pass capture_only: true for capture-only writes.

That makes Threadline useful in both of these shapes:

plug Threadline.Plug, actor_fn: &MyApp.Audit.current_actor/1

{:ok, %{post: post}} =
  Threadline.Audit.transaction(
    MyApp.Repo,
    [audit_context: audit_context, action: :post_created],
    fn ->
      case MyApp.Repo.insert(Post.changeset(%Post{}, attrs)) do
        {:ok, post} -> %{post: post}
        {:error, changeset} -> MyApp.Repo.rollback(changeset)
      end
    end
  )

Architecture layers

1. Capture substrate

PostgreSQL triggers write the durable audit rows. The core facts are AuditTransaction and AuditChange; the audit tables are the source of truth for row-level history.

This layer answers:

  • What actually changed?
  • In which transaction?
  • At what time?

It does not answer ownership or policy questions on its own.

2. Host-owned seams

Threadline intentionally keeps auth and tenancy on the host side.

  • Threadline.Plug owns request-path capture context.
  • Threadline.Job owns serialized job-path context.
  • Threadline.Integrations.* owns soft-loaded reference adapters.
  • threadline_operator_surface/2 owns the mount boundary for the optional operator UI.

The host decides who the actor is, which request context matters, and whether support access is admin-only or read-only.

3. Investigation layer

These are the read APIs most adopters use first:

Rule of thumb: use the eager helpers when the window is small enough to read in one shot, and the paged helper when it is not.

4. Operator surface

The /audit surface is optional and lives in-tree for now. It gives you a host-mounted UI for the same investigation questions as the library APIs and Mix tasks.

That means the library and the UI answer the same questions:

The SaaS Builder's JTBD Map

If you are dropping Threadline into your SaaS, you are hiring it to do four very specific jobs.

Job 1: The "Silent Witness" (Compliance & Baseline)

  • The Scenario: A SOC2 auditor wants proof of data lineage, or a customer is screaming, "I never deleted that invoice!" You need to know that no matter what happens, the truth is recorded.
  • The Flow: You run mix threadline.gen.triggers to attach PostgreSQL triggers to your tables. You don't touch your Elixir contexts. Even if a junior dev opens an iex shell and runs MyApp.Repo.delete_all(), the triggers catch it.
  • The JTBD: "Give me an airtight, DB-level audit trail without forcing me to rewrite my application code to use special audit_insert functions."

Job 2: The "Who Did This?" (Attribution & Intent)

  • The Scenario: A database trigger only knows that the postgres database user modified a row. That’s useless for a SaaS. You need to know that user_id: 42 did it via the /billing/refund endpoint.
  • The Flow: You drop plug Threadline.Plug into your router and configure it to pull the current user from the session. Wrap business writes in Threadline.Audit.transaction/3 with :action when intent and correlation matter — the helper records semantic intent and links audit_transactions.action_id in the same database transaction as the row changes.
  • The JTBD: "Bridge the gap between physical database mutations and human application semantics so the logs actually make sense."

Job 3: The "3 AM Support Ticket" (Investigation & Ops UI)

  • The Scenario: A customer writes into Zendesk: "My dashboard looks weird since yesterday." Your ops team needs to figure out what state changed without bugging an engineer to write custom SQL.
  • The Flow: You mount the /audit LiveView dashboard in your host app. Support staff log in (using your app's existing auth). They filter the timeline by the customer's actor_id or the specific record_id and get a visual diff of exactly what fields changed, when, and by whom.
  • The JTBD: "Give my support and ops team a safe, read-only window into historical data state so they can unblock customers autonomously."

Job 4: The "Data Handoff" (Egress & Reporting)

  • The Scenario: Legal needs a CSV of every permission change in the Enterprise workspace over the last 30 days.
  • The Flow: Your ops team uses the filter form on the LiveView timeline, hits "Export", and downloads the results. Alternatively, you run mix threadline.export in your deployment console.
  • The JTBD: "Get the data out of the system in a standard, machine-readable format quickly and safely."

The library exists to make those personas overlap cleanly instead of forcing each one to build a different audit story.

What already landed around the core

The core capture + semantics + investigation loop is no longer standing alone. Threadline already ships the governance and operator-lifecycle layer that used to sit on the near-term roadmap:

  • Lifecycle & Pruning: retention admin with visible purge history and safe batched cleanup.
  • Async Export Delivery: queued or scheduled exports with status visibility, expiry cleanup, and backend-aware delivery seams.
  • Operator Ergonomics: saved views for repeated investigations without inventing a Threadline-owned auth model.

Those capabilities matter because they keep the investigation surface usable once an adopter moves beyond a single incident and starts operating Threadline as recurring support infrastructure.

The Line of Diminishing Returns

A great library knows what it isn't. Threadline hits the point of diminishing returns—and starts turning into bloated software—if we cross these lines:

  1. Becoming a SIEM: We are embedded infrastructure. We provide the facts. We will not build anomaly detection ML, chart builders, or pie-graph dashboards.
  2. Owning Auth/RBAC: Threadline relies on the host app to say "this user is an admin." We will not build user tables or role-permission matrices.
  3. UI-Based Policy Mutation: Security rules (like "don't log the passwords table" or "redact the ssn column") must live in code/config. We will not build a UI toggle to turn off logging, as that creates a vector for a rogue admin to disable logging, do something bad, and turn it back on.

Public API surface

If you only remember one thing, remember this grouping:

Write-side

Manual set_config + record_action/2 linkage recipes are deprecated — see Integration contracts § Audited write path via Threadline.Audit.

Read-side

Operator parity

The read-side APIs are the stable core. The operator surface and Mix tasks are convenience layers on top of the same investigation model.

Evolution so far

  • 0.1.x established the capture substrate and semantics layer.
  • 0.2.x hardened the query, continuity, and retention story.
  • 0.3.x brought the first serious host-integration seams and example-app path.
  • 0.4.x added the optional operator surface and its first investigation screens.
  • 0.5.0 tightened the breadth story: honest support lanes, shared host-owned auth seams, and a clearer optional-in-tree position for the UI.
  • 0.6.0 packaged the Evidence plane and Threadline.Audit.transaction/3 as the recommended audited write path.

That evolution matters because the library did not start as a product-console project. It became one as the investigation path matured.

Natural next work

The next chunks that feel naturally adjacent are:

  • broader first-party host adapters beyond the current Sigra reference path
  • deeper proof around how different hosts expose the support lane without widening Threadline into its own auth product
  • extraction pressure checks for whether a separate UI package is ever worth the maintenance cost
  • a threadline_web split only if objective extraction triggers show up

Those are the kinds of problems that usually belong in the next milestone once adopters start using the current surface for real.

Where to go next

Adoption discovery order:

  1. README — top-level map and version quick start
  2. This guide — mental model, formula, and flow
  3. Getting started with Phoenix SaaS §6 — canonical runnable Audit.transaction/3 snippet
  4. Domain reference — vocabulary and API routing
  5. Integration contracts — host seams and audited write path contracts
  6. Operator surface — mount, auth, and screens
  7. Support lanes and upgrade path — support matrix