PropertyDamage.ErrorOrigin (PropertyDamage v0.2.0)

View Source

Classifies test failures by their likely origin.

When a PropertyDamage test fails, the failure can come from:

  • SUT Error: A bug in the System Under Test (what we're trying to find)
  • Test Code Error: A bug in the test configuration (model, projections, commands, adapters)

This module analyzes failure reasons and stacktraces to determine the most likely origin, helping users quickly understand whether they need to fix their SUT or their test code.

Classification Categories

SUT Errors (High Confidence)

These failures almost certainly indicate bugs in the SUT:

  • :check_failed / :assertion_failed - Invariant violations
  • :poll_timeout - Temporal assertion timeout
  • :idempotency_violation - SUT not idempotent
  • :linearization_failed - Race condition detected

Test Code Errors (High Confidence)

These failures almost certainly indicate bugs in test code:

Ambiguous

These failures could be either:

  • Generic exceptions during assertion execution
  • Adapter errors (could be bad adapter code or SUT returning unexpected data)

Summary

Types

classification()
details()
origin()

Functions

classify(failure_reason, stacktrace \\ nil)

Classify a failure by its likely origin.

summary(map)

Get a human-readable summary of the classification.

sut_error?(arg1)

Returns true if the classification indicates a SUT error.

test_code_error?(arg1)

Returns true if the classification indicates a test code error.

Types

classification()

@type classification() :: %{origin: origin(), details: details()}

details()

@type details() :: %{
  reason: String.t(),
  evidence: map(),
  confidence: :high | :medium | :low
}

origin()

@type origin() :: :sut_error | :test_code_error | :unknown

Functions

classify(failure_reason, stacktrace \\ nil)

@spec classify(term(), list() | nil) :: classification()

Classify a failure by its likely origin.

Takes a failure reason and optional stacktrace, returns classification.

Examples

iex> ErrorOrigin.classify({:check_failed, :NonNegativeBalance, "Balance is -50"})
%{origin: :sut_error, details: %{reason: "Invariant violation", ...}}

iex> ErrorOrigin.classify({:adapter_error, %UndefinedFunctionError{...}}, stacktrace)
%{origin: :test_code_error, details: %{reason: "Missing callback", ...}}

summary(map)

@spec summary(classification()) :: String.t()

Get a human-readable summary of the classification.

sut_error?(arg1)

@spec sut_error?(classification()) :: boolean()

Returns true if the classification indicates a SUT error.

test_code_error?(arg1)

@spec test_code_error?(classification()) :: boolean()

Returns true if the classification indicates a test code error.