Maps extraction strings to byte spans in source text.
Mirrors upstream langextract's WordAligner (v1.6.0 + #485) semantics in
four phases over downcased word tokens:
- Occurrence DP — over the whole extraction list in model output
order: selects at most one exact occurrence per extraction, keeping
selections order-preserving and non-overlapping while maximizing total
matched tokens; ties prefer the earliest-ending chain, so repeated
mentions resolve to successive occurrences. Status
:exact. Extractions the DP cannot place fall through to the phases below. - Exact — the extraction's tokens appear contiguously in the source
(linear scan, first occurrence wins). Status
:exact. - Lesser — difflib-style block decomposition: if a matching block is
anchored at the extraction's first token, its source run grounds the
extraction (upstream
MATCH_LESSER). Blocks elsewhere in the extraction do not qualify. Status:fuzzy. - LCS fuzzy — for extractions sharing no token run at all, an LCS
subsequence match over lightly stemmed tokens, accepted when coverage
(matched / extraction tokens) ≥
:fuzzy_thresholdand density (matched / span length) ≥:min_density, preferring the tightest span. Status:fuzzy.
Known divergence from upstream: our fallthrough phases treat each leftover extraction standalone, while upstream reruns difflib over the concatenated tokens of all sibling extractions — see @known_divergences in aligner_parity_test.exs for the observable consequences.
Summary
Functions
@spec align(String.t(), [String.t()], keyword()) :: [LangExtract.Alignment.Span.t()]