Docket ships one source-checkout benchmark suite under bench/. It is not part
of the public API or Hex package contract and requires a dedicated PostgreSQL
database via DOCKET_BENCH_DATABASE_URL.
- Scorecard (
bench/postgres/scorecard.exs): system-level scenarios against the real supervised runtime, condensed into named 0–100 scores with invariant gates.
Scorecard
DOCKET_BENCH_DATABASE_URL=postgres://user@localhost:5432/docket_bench \
mix run bench/postgres/scorecard.exs -- --profile local
Options:
| Switch | Meaning |
|---|---|
--profile smoke|local|scale | workload size and score targets (default local) |
--only a,b | run a subset of scenarios |
--output PATH | artifact directory (default tmp/bench/postgres/scorecard/<run-id>/) |
--check | raise on an invariant violation, scenario error, or missing score; never gate on the numeric timing score |
--keep-schema | keep the scratch schema for inspection |
--seed N | deterministic interleave seed |
Each run creates an isolated scratch schema (docket_bench_<pid>_<uniq>),
installs the production migration into it, runs the selected scenarios
sequentially, prints one scorecard table, and writes manifest.json and
scorecard.json artifacts.
Scenarios and scores
The drain scenarios drive the real Docket.Runtime.Supervisor with the
Docket.Postgres backend (poll-only dispatch). Claim efficiency instead calls
RunStore.claim_due/3 directly with concurrent workers and intentionally
retains the resulting claims. Metrics come from durable rows, not telemetry;
for drain scenarios each run's wait is
finished_at − max(staged due time, runtime start), so seeding overhead
never counts as queue time. Scores are 0–100; the mapping from raw measurements
to a score is stated per scenario and every threshold below is an explicit
calibration knob in the profile.
| Metric | Scenario | Score definition |
|---|---|---|
| Throughput | drain N immediately-due one-step runs at fixed concurrency | 100 · min(1, runs_per_sec / target_runs_per_sec) |
| Concurrency scaling | same per-slot workload at increasing concurrency levels | 100 · clamp(E) where E = (T(c_max)/T(c_min)) / (c_max/c_min) |
| Claim efficiency | direct claim_due workers drain a frozen backlog (no runtime) | 100 · min(1, claims_per_sec / target_claims_per_sec) |
| Tenant fairness | one hot tenant seeds 60% of the backlog first; light tenants after | 100 · clamp(1 − light_p95_wait / drain_time) |
| Fast/slow fairness | fast cohort alone vs fast cohort behind a slow cohort | 100 at slowdown ≤ good, linear to 0 at ≥ bad |
| Surge resilience | steady arrivals at 40% of measured capacity plus a mid-window burst | 100 · min(1, ideal_recovery / measured_recovery) |
Claim-policy dimension
The claim-sensitive rows — claim efficiency, tenant fairness, and fast/slow
fairness — are expanded from the claim_policies registry
(bench/support/scorecard/config.ex) and carry the policy name in the
scenario column, e.g. 60% hot tenant @16 [legacy]. Each entry's config is
passed straight through as the runtime's claim_policy backend option. The
registry includes both Legacy and TenantFair. The TenantFair benchmark cap is
set above the largest frozen-backlog fixture because claim efficiency retains
claims by design; deterministic PostgreSQL tests, rather than scorecard rows,
remain the sticky-cap correctness evidence.
The tenant-fairness scenario requires tenant-scoped storage and therefore runs only with TenantFair. Claim efficiency and fast/slow fairness run with both registered policies. The formal Legacy comparison is the logical trace in the TenantFair Legacy separation, not a timing score.
Interpretation caveats:
- Throughput and claim-efficiency targets are machine-relative. They exist so a number becomes a trend line on one machine; comparing scores across hardware is meaningless. Ratio-based scores (scaling, fairness, surge) are dimensionless and travel better.
- Fairness scores depend on the selected claim policy. Keep comparisons on the same database and machine, and treat them as regression signals rather than release evidence.
- Claim efficiency measures the raw claim path under concurrent callers. Production serializes admission through one dispatcher per instance, so this is a ceiling probe, not a production simulation.
- Latency in drain scenarios is queue plus service time by construction (all runs due at once), which is what the fairness comparisons need.
Invariants
Drain scenarios assert, via SQL on the scratch schema, no duplicate active
claim tokens, no active claims after drain, exact terminal accounting, no
stranded or poisoned runs, unique event sequence per run, and exactly one
terminal event per run. Claim efficiency has a separate frozen-backlog
invariant: every seeded run remains nonterminal and claimed exactly once, with
no poison or terminal events. Any applicable violation forces the scenario's
score to zero as GATED, and --check raises. A fast result can never outrank
a wrong one.
Profiles
smoke finishes in well under a minute and is the optional local plumbing
check (--profile smoke --check); it is not a v0.1 CI release gate. local is
the default developer profile; scale grows backlogs roughly an order of
magnitude for dedicated runs. Exact knobs live in
bench/support/scorecard/config.ex.