@spec classify([gap()], MapSet.t(String.t())) :: [gap()]

Classifies each gap's :status (see the moduledoc's "Classifying tier-5 gaps" section and gap_status/0): :gap, :excluded, or :suite_local.

builtin_names is the set of function names the classifier treats as real builtins - in practice the caller passes Predicator.Evaluator.merge_functions([])'s keys, so this module never hardcodes "what is a builtin" itself. A gap whose pattern is not a call:<name> pattern (i.e. not tier 5) is :gap unless it is the documented opcode exclusion relative_date.

Examples

iex> gaps = [
...>   %{tier: 5, tier_name: "functions", pattern: "call:Math.abs", suite_count: 3, corpus_count: 0},
...>   %{tier: 5, tier_name: "functions", pattern: "call:Date.now", suite_count: 6, corpus_count: 0},
...>   %{tier: 5, tier_name: "functions", pattern: "call:boom", suite_count: 2, corpus_count: 0},
...>   %{tier: 4, tier_name: "rich types", pattern: "relative_date", suite_count: 4, corpus_count: 0}
...> ]
iex> Predicator.Conformance.Coverage.classify(gaps, MapSet.new(["Math.abs", "Date.now"]))
...> |> Enum.map(& &1.status)
[:gap, :excluded, :suite_local, :excluded]

@spec corpus_pattern_frequencies([map()]) :: pattern_frequencies()

Computes pattern frequencies from decoded corpus case maps (the shipped conformance/corpus/tier-*.json cases, already JSON.decode/1-ed).

Reads each case's "instructions" field the same way suite_pattern_frequencies/1 reads a compiled instruction list - the tagged-value codec only affects operand values, never the opcode strings this counts.

Examples

iex> cases = [%{"instructions" => [["load", "a"], ["lit", 1], ["compare", "GT"]]}]
iex> Predicator.Conformance.Coverage.corpus_pattern_frequencies(cases)
%{"load" => 1, "lit" => 1, "compare:GT" => 1}

@spec diff(pattern_frequencies(), pattern_frequencies()) :: [gap()]

Diffs suite pattern frequencies against corpus pattern frequencies.

Returns every pattern the suite hits more than the corpus does (corpus absence counts as 0), grouped implicitly by carrying each entry's tier - format_report/1 does the grouping for display. A pattern whose opcode is not in Predicator.Instructions.opcodes/0 (should not happen for anything that compiled, but kept defensive) is skipped rather than crashing the report.

Sorted by tier, then by descending suite count, then by pattern name, so the most-exercised gap in the lowest tier sorts first - the tier a sibling is most likely to be working on right now.

Examples

iex> suite = %{"compare:GT" => 5, "compare:STRICT_EQ" => 2, "lit" => 10}
iex> corpus = %{"compare:GT" => 5, "lit" => 3}
iex> Predicator.Conformance.Coverage.diff(suite, corpus)
[
  %{tier: 1, tier_name: "core", pattern: "lit", suite_count: 10, corpus_count: 3},
  %{tier: 1, tier_name: "core", pattern: "compare:STRICT_EQ", suite_count: 2, corpus_count: 0}
]

@spec documented_exclusion_functions() :: MapSet.t(String.t())

The builtin function names documented as non-deterministic exclusions in conformance/README.md's "Functions excluded from the coverage rule" section - Date.now and Math.random. classify/2 marks a call:<name> gap whose name is in this set as :excluded rather than :gap, the same treatment relative_date gets as an opcode-level exclusion.

Examples

iex> Predicator.Conformance.Coverage.documented_exclusion_functions() |> Enum.sort()
["Date.now", "Math.random"]

@spec extract_sources(String.t()) :: [String.t()]

Extracts literal source strings from one file's Elixir source.

Scans the AST (not a regex over the text) for calls to Predicator.evaluate/2, Predicator.evaluate/3, or Predicator.compile/1 whose first argument is a literal string. A file that fails to parse returns [] rather than raising - this is a best-effort scan over test files, not a compiler.

Examples

iex> Predicator.Conformance.Coverage.extract_sources("Predicator.evaluate(\"a > 1\", %{})")
["a > 1"]

iex> Predicator.Conformance.Coverage.extract_sources("Predicator.compile(\"x\")")
["x"]

iex> Predicator.Conformance.Coverage.extract_sources("Predicator.evaluate(source, %{})")
[]

iex> Predicator.Conformance.Coverage.extract_sources("not valid elixir (")
[]

@spec format_report([gap()]) :: String.t()

Formats a list of gaps (as returned by diff/2, typically classified by classify/2 first) as a human-readable report, grouped by tier so it is actionable one tier at a time.

A gap's :status (absent counts as :gap) controls how it is shown: :gap prints plainly, :excluded gets an inline note instead of appearing as an unqualified corpus: 0 row, and :suite_local entries are pulled out of every tier and moved to a trailing labelled section rather than being silently dropped - this is a heuristic instrument, and a reader who remembers writing a test against one of these names should see where it went, not wonder if the scan missed it.

Examples

iex> Predicator.Conformance.Coverage.format_report([])
"No coverage gaps found: every pattern the suite's extracted sources exercise\nappears at least as often in the corpus.\n"

@spec suite_pattern_frequencies([String.t()]) :: pattern_frequencies()

Computes pattern frequencies for a list of extracted source strings.

Each source is compiled through the real Predicator.compile/1. A source that fails to compile contributes nothing - the extraction in extract_sources/1 is approximate, so an uncompilable "source" is expected (e.g. a string that merely looked like one).

Examples

iex> Predicator.Conformance.Coverage.suite_pattern_frequencies(["1 > 2", "1 > 2", "a == b"])
%{"compare:GT" => 2, "compare:EQ" => 1, "lit" => 4, "load" => 2}