Differential testing for comparing multiple implementations.

Differential testing runs the same command sequences against multiple targets (adapters) and compares results. This enables:

• Oracle testing: Compare SUT against a reference implementation
• Performance comparison: Compare latency/throughput across implementations
• Regression testing: Compare old vs new versions
• Migration validation: Compare legacy vs new systems

Basic Usage

# Oracle testing (correctness comparison)
PropertyDamage.Differential.run(
  model: MyModel,
  targets: [
    {OracleAdapter, role: :reference},
    {SUTAdapter, name: "new-impl"}
  ],
  compare: :correctness
)

# Performance comparison
PropertyDamage.Differential.run(
  model: MyModel,
  targets: [
    {ImplA, name: "redis-backend"},
    {ImplB, name: "postgres-backend"}
  ],
  compare: :performance
)

Same Adapter, Different Configurations

A key use case is comparing the same adapter with different configurations:

PropertyDamage.Differential.run(
  model: MyModel,
  targets: [
    {HTTPAdapter, role: :reference, opts: [base_url: "https://prod.example.com"]},
    {HTTPAdapter, name: "staging", opts: [base_url: "https://staging.example.com"]}
  ],
  compare: :correctness
)

Time-Separated Execution

Run against one system now, save results, compare later:

# Save baseline
PropertyDamage.Differential.run(
  model: MyModel,
  targets: [{ProdAdapter, name: "v2.3"}],
  compare: :performance,
  export_to: "baselines/v2.3.json"
)

# Later, compare against baseline
PropertyDamage.Differential.run(
  model: MyModel,
  targets: [{ProdAdapter, name: "v2.4"}],
  compare: :performance,
  baseline: "baselines/v2.3.json"
)

Execution Modes

• :interleaved - Execute commands round-robin across targets (default for correctness)
• :sequential - Execute full sequence on each target (default for performance)

When using baseline:, execution is implicitly sequential.

Equivalence Strategies

For correctness comparison:

• :exact - Results must be identical (default)
• :structural - Ignore common non-deterministic fields (id, timestamps)
• Custom function - fn ref_result, target_result -> boolean