Operator runbooks

This guide is the RUN-01 procedural companion to guides/telemetry.md. That file owns the ops event catalog for [:accrue, :ops, :*] — Accrue does not duplicate that table here. Use this document for ordered triage, Oban queue defaults, expanded Stripe verification, and the four mini-playbooks where sequence matters.

Library vs host: Accrue ships workers and suggested queue names; your host application configures and starts Oban (queues, concurrency, pruning). Queue names below are defaults Accrue documents in code — you may remap them in host config; treat symptoms and checks as patterns, not hard-coded production names.

Oban queue topology

Queue names are host-configurable; the table lists Accrue’s documented defaults from use Oban.Worker in accrue/lib today.

Stripe verification pattern

Use a two-layer mental model whenever Stripe is involved:

1. Accrue layer (operational): local rows (accrue_* tables), telemetry and operation_id, foreign keys and Stripe ids stored by Accrue (cus_*, sub_*, pi_*, Connect account ids, etc.). This is application state for billing workflows — useful for triage, not a substitute for Stripe’s financial records. For customer billing portal failures, correlate [:accrue, :billing, :billing_portal, :create] :stop / :exception latency with accrue.customer.id and operation_id per telemetry.md — do not paste %Accrue.BillingPortal.Session{} inspect output into tickets. For Stripe Checkout sessions created via Accrue.Billing.create_checkout_session/2, use the same pattern on [:accrue, :billing, :checkout_session, :create], confirm whether the host runs Accrue.Processor.Fake vs live Stripe, and read the PII-safe metadata contract at telemetry.md#billing-checkout-session-create — do not paste session URLs or client_secret values into tickets.
2. Stripe layer (verification): confirm each issue against the Stripe resource type + id using canonical documentation (e.g. Webhooks, Testing webhooks, Billing meter events) and functional Dashboard paths (e.g. Developers → Webhooks → event deliveries) rather than brittle deep links.

For finance and tax reporting, use Stripe Dashboard / reporting products as your source of truth; Accrue focuses on state, webhooks, and replay in your app.

Mini-playbook: [:accrue, :ops, :webhook_dlq, :dead_lettered]

1. Confirm scope: identify event_id / processor_event_id from telemetry or admin (do not paste full webhook payloads or secrets into tickets).
2. Inspect the accrue_webhook_events row and last error; decide fix vs replay before mutating data.
3. Check Oban for Accrue.Webhook.DispatchWorker on :accrue_webhooks (see Oban queue topology); ensure the host queue is running and not wedged.
4. If replay is required, prefer admin-gated or documented replay flows; use dry-run when available — avoid destructive deletes from this path.
5. Cross-check the same event type in Stripe via Developers → Webhooks → recent deliveries (Webhook docs).
6. After fix, enqueue or allow retry; watch [:accrue, :ops, :webhook_dlq, :replay] and related metrics for confirmation.

If the dead-lettered row is Braintree-sourced or tied to local portal checkout:

7. Confirm whether the failed row should reduce into accrue.portal.checkout.completed or a normalized subscription/invoice event before replaying it.
8. Fix host-local causes first: portal_base_url, portal_mount_path, auth/session continuity, or Hosted Fields readiness.
9. Replay the persisted row only after the mounted path is healthy; Braintree recovery is local projection convergence, not an upstream hosted checkout retry.

Mini-playbook: [:accrue, :ops, :events_upcast_failed]

1. Record event_id, type, and schema_version from the ops metadata (identifiers only).
2. Determine whether a deployed upcaster is missing vs bad persisted data — do not replay until the schema path is understood.
3. Inspect Accrue.Events / event storage per your host (see catalog row in telemetry.md); align with code version in the running release.
4. Verify Oban or inline retry behavior will not amplify a bad version skew; pause automated replay if unsure.
5. Queue topology for indirect jobs: see Oban queue topology if downstream dispatch is involved.
6. Validate against Stripe only if the failing payload is a Stripe-sourced event; use Event object docs for shape, not as ledger truth.

Mini-playbook: [:accrue, :ops, :meter_reporting_failed]

Always read the contract (when the tuple fires and what each source means) at telemetry.md#meter-reporting-semantics before changing alert thresholds—this runbook is procedure only.

1. Read source (:sync, :webhook, :reconciler) plus meter_event_id / event_name from metadata (identifiers only—no raw payloads).
2. Load the matching accrue_meter_events row and note stripe_status, stripe_error, and timestamps so you know whether the failure epoch is already terminal.

:sync (host request path)

1. Correlate with the host request or job that called Accrue.Billing.report_usage/3 in the same transaction window; inspect logs around Accrue.Billing.MeterEventActions for processor errors surfaced synchronously.
2. Fix configuration or upstream Stripe errors, then retry the host operation with a fresh operation_id only when the business case requires a new attempt—idempotent replays should converge on the stored terminal row.

:reconciler (Oban :accrue_meters)

1. Inspect Oban jobs for Accrue.Jobs.MeterEventsReconciler on :accrue_meters (Oban queue topology); confirm the queue is running and not wedged behind retries.
2. After correcting Stripe meter setup or credentials, allow the reconciler to dequeue; watch [:accrue, :ops, :meter_reporting_failed] and default metrics for confirmation.

:webhook (meter error report path)

1. Trace the event through accrue_webhook_events into Accrue.Webhook.DefaultHandler and the async Accrue.Webhook.DispatchWorker path; verify signature + dispatch health before mutating rows (Oban queue topology).
2. Resolve the upstream Stripe meter error, then replay or wait for the next reconciler pass; confirm the row leaves terminal failed only when business logic intentionally clears it.

Shared verification (all sources):

3. Confirm API keys and Stripe meter configuration for the environment (no key material in logs).
4. Cross-check Stripe usage reporting with Metered billing — operational alignment, not accounting close.
5. After code or config fix, allow reconciler retry where applicable; watch ops counters and host metrics.

Mini-playbook: Braintree metered renewal and settlement recovery

These steps apply to the Braintree-local metering tuples documented in telemetry.md. The ordering matters because Accrue's local invoice ledger is canonical and Braintree is settlement-only in this flow.

[:accrue, :ops, :metered_renewal_stale_repaired]

1. Confirm the affected metered_renewal_id maps to a subscription period that should already have advanced.
2. Inspect the corresponding subscription in Braintree and verify the cycle actually renewed; the backstop should mirror webhook truth, not invent renewal windows.
3. Check Accrue.Jobs.MeteredRenewalReconciler and Accrue.Jobs.ProcessMeteredRenewal on :accrue_meters (Oban queue topology) so the repaired window continues into local invoice authoring and settlement.
4. If the renewal only became visible after webhook backlog or replay work, pair this tuple with [:accrue, :ops, :webhook_dlq, :replay] so the replay trail and the stale-window repair tell one story.

[:accrue, :ops, :metered_missing_definition]

1. Inspect the renewal window and its unmatched meter events; identify which event_name rows lack a local meter definition.
2. Add or repair the missing definition so future windows classify those events explicitly.
3. Replay the same renewal window after the definition exists; do not create ad-hoc manual charges that bypass the local invoice decomposition.

[:accrue, :ops, :metered_charge_awaiting_payment_method]

1. Repair or replace the customer's default vaulted payment method.
2. Confirm the local invoice for that renewal window is still the correct settlement target.
3. Replay the same renewal window so Accrue reuses the existing charge unit instead of creating a second Transaction.sale.
4. If checkout completion is still ambiguous for the same customer, verify whether accrue.portal.checkout.completed already persisted locally before creating any manual recovery plan.

[:accrue, :ops, :metered_charge_failed_exhausted]

1. Confirm the failure class and the current local invoice state before retrying anything.
2. Decide whether to retry, write off, or pair the failed renewal with a later operator-approved recovery step.
3. Preserve the original failed attempt trail; do not delete the renewal or charge-attempt rows to force a clean slate.

Mini-playbook: [:accrue, :ops, :revenue_loss]

1. Capture reason, subject_type, subject_id, and currency amounts from telemetry (aggregates / IDs only — no customer narrative in shared logs).
2. Triage Accrue rows (invoice, credit note, adjustment) that triggered the signal; avoid manual balance edits without a controlled procedure.
3. Check related async work on :accrue_reconcilers and :accrue_webhooks if the loss correlates with webhook or fee reconciliation (Oban queue topology).
4. In Stripe, locate the same business object (charge, refund, dispute) via Dashboard search or list filters; use Balance transactions categories as reference for classification, not as instructions to reproduce Sigma in-app.
5. Document outcome in your ticketing system; escalate finance questions on Stripe’s side, not via Accrue as a ledger substitute.

RUN-01 coverage

• Full ops tuple list and one-line first actions live under ## Operator runbooks (first actions) in telemetry.md — bookmark that table for every RUN-01 class, including :connect_account_deauthorized, :connect_payout_failed, :dunning_exhaustion, :charge_failed, :incomplete_expired, :pdf_adapter_unavailable, replay (:webhook_dlq, :replay), and prune (:webhook_dlq, :prune).
• This file adds depth for the four classic mini-playbooks above plus the Braintree metered-billing recovery sequence.

v1.59 multi-rail and offline runbooks

These procedures use the bounded actions in Accrue.Entitlements.Repair. The host authorizes every action and supplies the account, actor, reason, operation ID, and bounded target. Start with a read-only diagnostic and a dry run where the action supports it. Record only the scenario/runbook ID, safe correlation, actor, reason, and before/after revision. Stop when authorization, target identity, provider health, or the post-action convergence check is uncertain.

No procedure here may automatically reconstruct an account, transfer or merge ownership, refund, cancel, migrate, prorate, or otherwise mutate provider or financial state. Escalate those decisions to the host's approved finance or product process.

V159-RUN-MISSED-NOTIFICATION — missed notification recovery

1. Confirm apple_purchase_to_web_login or the affected scenario ID, account revision, and safe lineage correlation from the bounded diagnostic.
2. An authorized operator records the reason and runs a dry-run of retry_missed_notification against that one lineage/environment target.
3. If the dry run is correct, queue the named action once and wait for the diagnostic's next revision; stop on an ambiguous lineage or provider error.
4. Record the post-convergence revision. Do not replay raw notification data or alter a provider purchase.

V159-RUN-CURSOR — history cursor recovery

1. Confirm the affected lineage, environment, cursor age, and the interrupted_resume or related scenario ID.
2. An authorized operator dry-runs recover_history_cursor for the bounded lineage/environment target, then confirms the recorded reason.
3. Run the action only after confirming it resumes a known cursor; stop if a cursor would be guessed, rewound without evidence, or crosses environments.
4. Verify the new diagnostic revision and record the safe correlation.

V159-RUN-PROVIDER-OUTAGE — provider outage or rate limit

1. Confirm the provider freshness state, retry age, and safe correlation; do not copy provider payloads or credentials into the incident.
2. Hold new issuance when key or provider health is unsafe, preserve the last known canonical snapshot, and use bounded retry/backoff.
3. An authorized operator dry-runs retry_provider_check for one lineage and environment, then queues it when the provider has recovered.
4. Stop on repeated rate limits or an unknown response; record the post-action revision and open V159-WL-APPLE-API or V159-WL-STRIPE reassessment.

V159-RUN-OWNERSHIP-CONFLICT — ownership conflict containment

1. Confirm the bounded diagnostic state and survivor_grant scenario; retain the safe correlation and current revision.
2. An authorized operator dry-runs review_ownership_conflict and records the actor and reason.
3. Submit the review action only to contain and audit the conflict. Stop when a resolution would require account reconstruction, transfer, or merge.
4. Verify the review disposition and route any ownership decision to the host's approved product/support process.

V159-RUN-DUPLICATE — duplicate-charge escalation

1. Confirm duplicate_purchase_prevention, the bounded diagnostic, and a safe correlation; never place charge or transaction material in the ticket.
2. An authorized operator dry-runs escalate_duplicate_charge, then records the actor, reason, and current revision.
3. Submit only the escalation action and stop. It does not refund, cancel, adjust, or mutate a provider charge.
4. Verify the escalation disposition and hand finance resolution to the host's approved process. Reassess V159-WL-SUPPORT when the trigger recurs.

V159-RUN-DEVICE — stale or revoked device replacement

1. Confirm device_replacement or refund_revocation, the device state, safe correlation, and canonical revision from the bounded diagnostic.
2. An authorized operator dry-runs replace_revoked_device; it accepts no proof material, token, or credential.
3. Give the learner the literal next action to reconnect and register the replacement device. Stop if a stale state would expand access before reconnect.
4. Verify the new revision and retain only the redacted outcome.

V159-RUN-KEY-ROTATION — signing-key compromise or rotation

1. Treat a suspected compromise as a security incident. Record the affected key set identifier and safe correlation, never key material or signed proof.
2. Pause issuance where the host cannot verify safe key access, then have an authorized operator dry-run rotate_signing_keys for the bounded key-set target.
3. Run the named action after the host's key-management approval; stop on any algorithm fallback, unverifiable proof, or unknown key state.
4. Verify post-rotation convergence, run the relevant golden vectors, and open the dated V159-WL-SECURITY reassessment.

V159-RUN-BACKLOG — reconciliation backlog drain

1. Confirm backlog age, queue health, limit, and safe correlation from the bounded diagnostic; do not inspect or publish worker arguments.
2. An authorized operator dry-runs drain_reconciliation_backlog with a small, bounded limit and recorded reason.
3. Run the action only while provider and queue health are stable. Stop on a growing backlog, rate limit, or post-action convergence failure.
4. Verify the revision and backlog age after each bounded batch. Open V159-WL-HOST when host queue or resource changes are implicated.

V159-RUN-APPLE-INGRESS — Apple notification ingress triage

1. Read response-class trends and retain only a safe correlation. Treat 429 as temporary backpressure; compare it with the trusted deployment edge or shared rate policy before changing the host's single-node backstop.
2. Review quarantine growth, reconciliation age/backlog, and needs_repair in the authenticated diagnostic. Do not inspect or attach provider evidence, worker arguments, or failure detail to the incident.
3. Confirm the named :accrue_entitlements queue and reconciliation sweeper are healthy, then allow the existing bounded job to converge. Stop and escalate if backlog age grows, a dependency remains unavailable, or needs_repair persists.
4. Record the response trend, safe correlation, job state, and next action. This runbook does not authorize an automatic grant, ownership, finance, or provider mutation.

V159-RUN-APP-REVIEW, V159-RUN-PRIVACY, and V159-RUN-ROADMAP

• App Review: record the storefront change or rejection, review only supported in-app purchase/restore/management wording, obtain product/legal approval, and stop before publishing an external-purchase or runtime claim.
• Privacy: record the jurisdiction, DSR, retention proposal, or redaction finding; confirm bounded diagnostic access and remove sensitive evidence before release. Stop until the host's privacy owner signs the reassessment.
• Roadmap: when Android or a second adopter triggers Google Play work, create a dated proposal for a separate rail-policy research milestone. Do not add Play parity opportunistically.

See also

• guides/telemetry.md — ops catalog SSOT and Operator runbooks (first actions) table
• Accrue.Telemetry.Ops — emit/3 contract (lib/accrue/telemetry/ops.ex in the repo; published API on Hexdocs)
• Hexdocs path pattern: https://hexdocs.pm/accrue/ (pin the version to your mix.lock)