PropertyDamage.Diff (PropertyDamage v0.2.0)

Diff-based debugging for comparing passing and failing test runs.

When a test fails intermittently or after code changes, understanding exactly what changed between a passing and failing run is crucial. This module provides tools to capture execution traces and compare them.

Workflow

Capture traces during execution (passing and failing)
Compare traces to find divergence points
Display differences in a clear, actionable format

Usage

Compare Two Failure Reports

passing_report = PropertyDamage.run(model: M, adapter: A, seed: 123)
failing_report = PropertyDamage.run(model: M, adapter: A, seed: 456)

diff = PropertyDamage.Diff.compare_reports(passing_report, failing_report)
IO.puts(PropertyDamage.Diff.format(diff))

Compare Event Logs

diff = PropertyDamage.Diff.compare_events(passing_events, failing_events)

Compare States

diff = PropertyDamage.Diff.compare_states(state_before, state_after)

Output Example

╔══════════════════════════════════════════════════════════════╗
║                    EXECUTION DIFF                            ║
╚══════════════════════════════════════════════════════════════╝

Divergence at command 3: Withdraw(amount: 150)

┌─ Events ──────────────────────────────────────────────────────┐
│ PASS: [WithdrawSucceeded(balance: 50)]                        │
│ FAIL: [WithdrawFailed(reason: :insufficient_funds)]           │
└───────────────────────────────────────────────────────────────┘

┌─ State Before Command 3 ──────────────────────────────────────┐
│   balance: 200 → 50                                           │
│   + pending_withdrawals: 150                                  │
└───────────────────────────────────────────────────────────────┘

Summary

Types

command_diff()

diff_result()

event_diff()

state_diff()

trace()

Functions

compare_events(left_events, right_events)

Compare two event logs.

compare_reports(left, right)

Compare two failure reports to find differences.

compare_states(left_state, right_state)

Compare two state maps.

compare_traces(left, right)

Compare two execution traces.

create_trace(commands, events, states, result)

Create a trace from execution results.

format(diff, opts \\ [])

Format a diff result for display.

Types

command_diff()

@type command_diff() :: %{
  index: non_neg_integer(),
  left: struct() | nil,
  right: struct() | nil,
  status: :same | :different | :missing_left | :missing_right
}

diff_result()

@type diff_result() :: %{
  divergence_index: non_neg_integer() | nil,
  divergence_command: struct() | nil,
  command_diffs: [command_diff()],
  event_diffs: [event_diff()],
  state_diffs: [state_diff()],
  summary: String.t()
}

event_diff()

@type event_diff() :: %{
  command_index: non_neg_integer(),
  left_events: [struct()],
  right_events: [struct()],
  status: :same | :different | :extra_left | :extra_right
}

state_diff()

@type state_diff() :: %{
  command_index: non_neg_integer(),
  field: atom(),
  left_value: term(),
  right_value: term(),
  status: :same | :changed | :added | :removed
}

trace()

@type trace() :: %{
  commands: [struct()],
  events: [PropertyDamage.EventLog.Entry.t()],
  states: [map()],
  result: :pass | {:fail, term()}
}

Functions

compare_events(left_events, right_events)

@spec compare_events([PropertyDamage.EventLog.Entry.t()], [
  PropertyDamage.EventLog.Entry.t()
]) :: [
  event_diff()
]

Compare two event logs.

Groups events by command index and compares each group.

compare_reports(left, right)

@spec compare_reports(
  PropertyDamage.FailureReport.t() | map(),
  PropertyDamage.FailureReport.t() | map()
) :: diff_result()

Compare two failure reports to find differences.

Parameters

left - First report (typically passing or earlier)
right - Second report (typically failing or later)

Returns

A diff_result map with divergence information.

compare_states(left_state, right_state)

@spec compare_states(map(), map()) :: [state_diff()]

Compare two state maps.

Returns a list of field-level differences.

compare_traces(left, right)

@spec compare_traces(trace(), trace()) :: diff_result()

Compare two execution traces.

Parameters

left - First trace
right - Second trace

Returns

A diff_result map.

create_trace(commands, events, states, result)

@spec create_trace(
  [struct()],
  [PropertyDamage.EventLog.Entry.t()],
  [map()],
  :pass | {:fail, term()}
) ::
  trace()

Create a trace from execution results.

Parameters

commands - List of executed commands
events - Event log entries
states - List of state snapshots (one per command)
result - :pass or {:fail, reason}

format(diff, opts \\ [])

@spec format(
  diff_result(),
  keyword()
) :: String.t()

Format a diff result for display.

Options

:format - Output format (:terminal, :markdown, :json)
:max_value_length - Truncate values longer than this (default: 60)
:show_same - Show identical items (default: false)