LangExtract.Alignment.Aligner (LangExtract v0.6.0)

Copy Markdown View Source

Maps extraction strings to byte spans in source text.

Mirrors upstream langextract's WordAligner (v1.6.0 + #485) semantics in four phases over downcased word tokens:

  1. Occurrence DP — over the whole extraction list in model output order: selects at most one exact occurrence per extraction, keeping selections order-preserving and non-overlapping while maximizing total matched tokens; ties prefer the earliest-ending chain, so repeated mentions resolve to successive occurrences. Status :exact. Extractions the DP cannot place fall through to the phases below.
  2. Exact — the extraction's tokens appear contiguously in the source (linear scan, first occurrence wins). Status :exact.
  3. Lesser — difflib-style block decomposition: if a matching block is anchored at the extraction's first token, its source run grounds the extraction (upstream MATCH_LESSER). Blocks elsewhere in the extraction do not qualify. Status :fuzzy.
  4. LCS fuzzy — for extractions sharing no token run at all, an LCS subsequence match over lightly stemmed tokens, accepted when coverage (matched / extraction tokens) ≥ :fuzzy_threshold and density (matched / span length) ≥ :min_density, preferring the tightest span. Status :fuzzy.

Known divergence from upstream: our fallthrough phases treat each leftover extraction standalone, while upstream reruns difflib over the concatenated tokens of all sibling extractions — see @known_divergences in aligner_parity_test.exs for the observable consequences.

Summary

Functions

align(source, extractions, opts \\ [])

@spec align(String.t(), [String.t()], keyword()) :: [LangExtract.Alignment.Span.t()]