Temper.Analysis (temper v0.1.0)

Classifies recorded test outcomes into flaky tests and suspects.

The definition of flaky is deliberately conservative: a test is flaky when it both passed and failed on the same clean git SHA — the code did not change, the outcome did. Divergence that only shows up when dirty-tree runs are included lands in the secondary suspects bucket: uncommitted changes could explain it, so the confidence is lower.

What never counts: divergence across different SHAs (a commit may simply have broken and fixed the test), runs without a SHA, and non-run statuses (skipped/excluded/invalid). This trades recall for zero false positives — the report is meant to be trusted.

This module is part of Temper's functional core: records in, report data out, no side effects.

Summary

Types

evidence()

Per-SHA tallies backing a finding. dirty: true marks suspect evidence.

failure_signature()

A distinct failure signature observed in failing runs.

finding()

One flaky or suspect test, with its aggregated evidence.

report()

Functions

analyze(records, opts \\ [])

Analyzes records into a report of flaky tests and suspects.

Types

evidence()

@type evidence() :: %{
  sha: String.t(),
  dirty: boolean(),
  runs: pos_integer(),
  passed: non_neg_integer(),
  failed: non_neg_integer(),
  flake_rate: float(),
  failing_seeds: [integer()],
  failures: [failure_signature()]
}

Per-SHA tallies backing a finding. dirty: true marks suspect evidence.

failure_signature()

@type failure_signature() :: %{
  kind: String.t(),
  message: String.t(),
  hash: String.t()
}

A distinct failure signature observed in failing runs.

finding()

@type finding() :: %{
  module: String.t(),
  name: String.t(),
  file: String.t() | nil,
  line: pos_integer() | nil,
  async: boolean() | nil,
  runs: pos_integer(),
  passed: non_neg_integer(),
  failed: non_neg_integer(),
  flake_rate: float(),
  first_seen: String.t(),
  last_seen: String.t(),
  failing_seeds: [integer()],
  failures: [failure_signature()],
  evidence: [evidence()]
}

One flaky or suspect test, with its aggregated evidence.

report()

@type report() :: %{flaky: [finding()], suspects: [finding()]}

Functions

analyze(records, opts \\ [])

@spec analyze(
  [Temper.Record.t()],
  keyword()
) :: report()

Analyzes records into a report of flaky tests and suspects.

Options:

:min_runs — minimum runs on a SHA for its divergence to count as evidence (default 2). Raising it trades detection speed for confidence.

Both lists are sorted by flake rate (then run count), descending. A test with clean-SHA evidence appears only under :flaky, never under :suspects as well.