# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.9.0] - 2026-07-09

### Added

- **`Serializer.result_to_map/2` and `result_from_map/1`** — serialize the
  full `Result` (spans + errors + usage), not just span lists; the shape
  extends `to_map/2`'s with `"errors"` and `"usage"`. Error reasons are
  open terms, so they serialize as their `inspect/1` rendering — JSON-safe
  but one-way: loaded errors carry the rendered string.
  `chunk_error_to_map/1` is public alongside `span_to_map/1`.

- **Explicit stability tiers** — the docs now group modules as Core API
  (the SemVer contract), Advanced (public, best-effort), Providers, and
  Internal (no guarantees), and the README's new "Stability" section
  spells out the contract: the two entry points, which structs are stable
  to match on, which are public for matching but constructed via
  `template!/2`, and that `Client` is opaque. Internal-tier moduledocs
  carry the marker themselves, so a reader landing directly on an
  internal module's page sees its status.

### Changed

- **Deserialization validates field types, not just shape** —
  `Serializer.from_map/1` (and `load_jsonl/1`) now reject maps whose byte
  offsets or attributes have the wrong type, enforcing the `Span`
  invariant at the decode boundary: located spans carry non-negative
  integer offsets, `not_found` spans carry `nil`, attributes are a map.
  Previously such maps decoded into corrupted structs that crashed later
  in consumer offset arithmetic; now they fail fast as
  `{:error, :invalid_data}`. `result_from_map/1` applies the same checks
  to chunk errors.

- **Breaking: `ChunkError`, `ChunkResult`, and `Span` are promoted to
  `LangExtract.*`** (were `LangExtract.Pipeline.ChunkError`,
  `LangExtract.Pipeline.ChunkResult`, `LangExtract.Alignment.Span`) —
  they are contract structs consumers match on (`Result.spans`,
  `Result.errors`, `stream/4` events, `align/3`) and now carry top-level
  names like the rest of the contract surface (`Result`, `Extraction`).
  Alignment/pipeline machinery (`Aligner`, `Tokenizer`, `Parser`) stays
  namespaced. Migration: drop the middle segment from aliases and struct
  patterns — `LangExtract.Alignment.Span` → `LangExtract.Span`,
  `LangExtract.Pipeline.ChunkError` → `LangExtract.ChunkError`.

## [0.8.0] - 2026-07-08

### Added

- **Programmatic usage: `Result.usage` and `ChunkResult.usage`** — token
  totals from the return value, no telemetry handler required:
  `result.usage.output_tokens` after a `run/4`, per-chunk on every
  `ChunkResult` stream event. `nil` when the provider reported no usage
  block; with partial chunk failures the totals cover the chunks that
  reported. Telemetry emission is unchanged — the same numbers now flow
  both ways.

- **`Span.located?/1`** — the documented guard for offset arithmetic:
  `true` for `:exact`/`:fuzzy` spans (offsets present), `false` for
  `:not_found` (offsets `nil`). `Enum.filter(spans, &Span.located?/1)`
  replaces every consumer hand-rolling the status check.

### Changed

- **Template attribute keys normalize to strings at construction** —
  map-authored example attributes (`%{kind: "port"}`) now produce the
  same string-keyed shape the wire format decodes (`%{"kind" => "port"}`),
  so prompt-rendered examples and parsed output never differ by key type.
  Ready-made `Extraction` structs pass through unchanged.
- **Breaking: `template/2` is renamed `template!/2`; `template/2` now
  returns tagged tuples** — the raising constructor gets the bang the
  stdlib convention demands (`URI.new/new!`), and the un-suffixed name
  becomes the non-raising twin for runtime task definitions:
  `{:ok, Template.t()} | {:error, ArgumentError.t() | ValidationError.t()}`.
  Migration: append `!` to existing calls. Wrong-typed fields (non-string
  `text`/`class`, non-list `extractions`, non-map `attributes`, non-map
  examples) also return the tagged `ArgumentError` — previously they
  raised `Protocol.UndefinedError` or `FunctionClauseError` deep in
  normalization, undermining the non-raising contract.
- **Breaking: `Provider.infer/2` returns `%Provider.Response{}`** (was a
  bare `{:ok, text}`) — the struct carries `text` plus `usage`
  (input/output token counts, `nil` when the API omits them). Providers
  were already parsing usage and discarding it into telemetry; now it
  reaches callers programmatically. Only affects direct callers of the
  provider layer and third-party `Provider` implementations — `run/4`,
  `stream/4`, and the Runner are unchanged by this entry.
- **Breaking: `run/4` returns `{:ok, %LangExtract.Result{}}`** (was
  `{:ok, {spans, chunk_errors}}`) — in both `LangExtract.run/4` and
  `Runner.run/4`. The struct binds by key, so future fields (usage,
  timing) can be added without breaking consumer matches; the positional
  tuple could never grow. Migration is mechanical:
  `{:ok, {spans, errors}}` → `{:ok, %LangExtract.Result{spans: spans,
  errors: errors}}`.

- **Tokenizer adopts upstream's letter/digit/symbol-run splitting** —
  possessives and contractions split at the apostrophe (`Tooke’s` →
  `Tooke`·`’`·`s`), numbers split at separators, and symbol runs are
  same-character tokens (`...` is one token). This closes the documented
  contraction-tokenization divergence: bare-name extractions against
  possessive source mentions now ground `exact` instead of `not_found`.
  Measured effect (2026-07-07 baseline): ner not_found 49 → 0 and fuzzy
  44 → 3 — exceeding upstream's own 1 and 45 — with dialogue fuzzy
  15 → 10; see benchmark/BASELINE.md. The `smart_quote_contraction`
  parity case now asserts
  upstream's real result. Chunker sentence rules mirrored to upstream's
  `find_sentence_range`: terminator-run matching (`...` ends sentences),
  abbreviation pairing (`"Dr" <> "."`), and break-unless-lowercase after
  newlines (lines opening with quotes or digits now break). Alignment
  offsets for spans involving contractions/possessives may shift; chunk
  boundaries on decimal-heavy text may differ.

### Fixed

- **`Runner` limiter refill no longer discards fractional tokens** — each
  refill reset the bucket's clock, dropping up to one token's worth of
  elapsed time per refill event; at low `:rpm` the under-delivery was
  proportionally large. Partial tokens now carry into the next refill.

## [0.7.0] - 2026-07-06

### Added

- **`LangExtract.Runner`** — a caller-owned, supervised extraction runner
  with a shared request budget. Place it in your supervision tree with a
  client, `:rpm`, `:max_in_flight`, and `:chunk_retries`; `Runner.run/4`
  and `Runner.stream/4` mirror the standalone APIs but schedule every
  chunk request through one Limiter (token-bucket RPM + in-flight cap),
  so concurrent callers cannot jointly exceed the budget and a single
  429 pauses all admission until the server's `retry-after` deadline.
  The runner owns its retry policy (Req's transient retry is disabled
  inside it): 429 waits never consume the per-chunk retry budget,
  5xx/transport failures take jittered backoff and do, other errors
  fail fast. Stream delivery is bounded — at most `:buffer` undelivered
  results — so a slow consumer throttles admission instead of growing a
  mailbox. `Runner.stream_corpus/4` runs an enumerable of `{id, source}`
  pairs through the same budget. Shutdown drains gracefully: in-flight
  requests get `:drain_timeout` to finish and deliver; unstarted chunks
  come back as `%ChunkError{reason: :drained}`. In runner mode every
  failure is per-chunk; there is no abandon-the-document error path.
  New guide: "Running in Production". New telemetry:
  `[:lang_extract, :limiter, :wait]` (duration, blocking reason, limiter)
  and `[:lang_extract, :chunk, :retry]` (attempt, reason, limiter).
- **`LangExtract.stream/4`** — lazy stream of per-chunk results
  (`{:ok, %Pipeline.ChunkResult{}}` | `{:error, %Pipeline.ChunkError{}}`)
  in completion order, so first spans arrive while later chunks are still
  extracting. `run/4` is now a collect-and-sort consumer of the same
  pipeline — one code path, contract unchanged. Stream mode keeps every
  failure per-chunk (a timed-out chunk is an error event with its byte
  range; survivors keep flowing), where `run/4` retains its
  abandon-the-document `{:error, {:task_exit, reason}}` contract.
  Document telemetry fires at consumption: `:start` on first demand,
  `:stop` at stream end, including early halts.
- **`LangExtract.template/2`** — the front door for building templates:
  accepts plain maps with string or atom keys (JSON-loaded task definitions
  work verbatim), normalizes into structs, and validates examples against
  the production aligner at construction — misaligned examples raise. Pass
  `validate: false` to skip.

### Changed

- **Oversized sentences hard-split at token boundaries** — text without
  sentence boundaries (logs, minified content) previously became one
  whole-document chunk, defeating `:max_chunk_chars`. Sentences past the
  budget now pre-split into token-boundary fragments (byte offsets exact;
  a single token longer than the budget stays whole), matching upstream's
  `ChunkIterator` — verified chunk-count parity on a boundary-free fixture.
- **Runner 429 retries are bounded** — absent a `retry-after` header the
  global pause now escalates exponentially (capped at 30s), and after
  `:rate_limit_retries` 429s on one chunk (default 10) the chunk fails
  with the rate-limit error instead of retrying forever. `retry-after`,
  when present, still sets the pause; rate-limit waits still never
  consume `chunk_retries`.
- **Breaking: decoding is JSON-only — YAML support removed entirely** —
  `WireFormat.normalize/1` no longer falls back to a YAML parser, the YAML
  repair machinery is deleted, and the `yaml_elixir`/`yamerl` dependencies
  are dropped. JSON has been the wire format since 0.6.0 and two of three
  providers constrain JSON at the API level; the tolerance path's
  justification ended with the format switch (audit simplicity finding).
  If you re-parse stored raw responses from the 0.4–0.5 YAML era through
  `extract/3`, convert them to JSON first.
- **Breaking: 429 errors carry the retry-after deadline** —
  `{:error, :rate_limited}` is now `{:error, {:rate_limited, ms | nil}}`;
  the runner's global backoff needs the server's deadline and it only
  exists on that response. Update any code matching on `:rate_limited`.
- **Breaking: `Prompt.Template` is now `LangExtract.Template`; `ExampleData`
  is now `Template.Example`** — template data is core (same promotion
  `Extraction` got in 0.4.0), and the example struct is a subordinate type
  nested in its owner. Construct via `LangExtract.template/2`; the structs
  remain public for pattern matching.
- **`req` constraint tightened to `~> 0.6.0`** (was `~> 0.6`) — pre-1.0
  minors are breaking by convention, so the constraint states what CI
  actually proves.
- **Specs name the provider error union** — the new
  `t:LangExtract.Provider.error/0` type covers every error
  `c:LangExtract.Provider.infer/2` can return; the provider modules,
  `Runner.Request.infer/4`, and `map_response/2` use it instead of
  `{:error, term()}`, and `run/4`'s error spec narrowed to
  `{:error, {:task_exit, term()}}`. Spec-only — no runtime change.

## [0.6.0] - 2026-07-05

### Security

- **Providers no longer follow HTTP redirects** — Req strips only the
  standard `authorization` header on cross-host redirects, so Claude's
  `x-api-key` would have been forwarded to a redirect target. LLM APIs
  never legitimately redirect these POSTs; a 3xx now surfaces as
  `{:error, {:api_error, status, body}}`. Re-enable via
  `req_options: [redirect: true]` if you proxy through something that
  redirects.

### Changed

- **Prompts adopt upstream's Q/A scaffold** — `Examples` heading, `Q:`/`A:`
  pairs, and a trailing bare `A:` answer primer, mirroring langextract's
  `QAPromptGenerator`. Measured effect: ~20% fewer output tokens (reduced
  adaptive-thinking spend), no alignment cost.
- **`req` constraint tightened to `~> 0.6`** — the previous `~> 0.5`
  admitted pre-1.0 minors the test suite has never run against.
- **`WireFormat.normalize/1` parses JSON first** — the strict, fast parser
  handles the (now default) JSON responses; the YAML parser and its repair
  pass remain as the tolerance path for models that answer in YAML.
- **Wire format is now JSON (was YAML)** — `WireFormat.format_extractions/1`
  emits fenced dynamic-key JSON, matching upstream's default; the `ymlr`
  dependency is dropped. Decided by a corpus A/B under the new scaffold:
  zero chunk errors across 440 quote-dense dialogue chunks, better dialogue
  alignment (17 fuzzy / 0 not_found vs 68 / 2), and 25% fewer ner output
  tokens. Decoding is format-agnostic — `WireFormat.normalize/1` accepts
  JSON and YAML responses alike and keeps the YAML repair machinery.

### Fixed

- **`Serializer.from_map/1` validates extraction entries** — entries
  missing `"text"` or carrying a non-string `"class"` now return the
  promised `{:error, :invalid_data}` instead of producing malformed spans.
  Class-less spans (from `align/3`) still round-trip.
- **Chunk task timeouts now return the documented error tuple** — the
  `{:error, {:task_exit, reason}}` shape promised by `run/4` was
  unreachable: `Task.async_stream`'s default `on_timeout: :exit` crashed
  the calling process instead. `on_timeout: :kill_task` makes a timed-out
  chunk surface as the documented infrastructure-failure return.

### Added

- **Telemetry** — `[:lang_extract, :request]`, `[:lang_extract, :chunk]`,
  and `[:lang_extract, :document]` spans (`:telemetry` is now an explicit
  dependency). Request `:stop` events carry input/output token counts
  normalized across all three providers; the benchmark runners record them
  as per-document `usage` blocks with per-request latency.

## [0.5.0] - 2026-07-05

### Added

- **Alignment tuning options** — `:min_density` (LCS token-density floor,
  default 1/3), `:accept_lesser` (toggle prefix matching), and
  `:exact_algorithm` (`:dp` | `:first_occurrence`), accepted by
  `LangExtract.run/4`, `extract/3`, and `align/3`.
- **"Alignment and Spans" hexdocs guide** — span semantics, byte-vs-character
  offsets (`binary_part`, not `String.slice`), the four aligner phases, and
  tuning. The hexdocs sidebar now groups modules by layer, and the
  `align`/`extract` examples run as doctests.

### Changed

- **Aligner ported to upstream langextract v1.6.0 semantics** — the fuzzy
  phase's frequency-overlap sliding window is replaced by upstream's
  difflib-style lesser prefix match plus an LCS dynamic program over
  lightly stemmed tokens, gated by coverage (`:fuzzy_threshold`) and density
  (`:min_density`). Behavior is pinned by differential fixtures generated
  from the upstream aligner. On the dialogue benchmark this took the exact
  rate from 77.5% to 93.7% — identical to Python's 93.7% on the same corpus.
- **Every prompt now demands verbatim spans** — `Prompt.Builder` appends a
  standing instruction requiring extractions to be exact source substrings
  and an empty `extractions: []` on contentless passages. Benchmarked:
  without it, dialogue runs produced dozens of few-shot echoes and stitched
  paraphrases that could not be aligned.
- **Repeated mentions now ground to successive occurrences** — the aligner
  gained a phase-0 monotonic occurrence DP (port of upstream #485): over the
  extraction list in model output order, it selects at most one exact
  occurrence per extraction, order-preserving and non-overlapping,
  maximizing matched tokens. Previously every within-chunk repeat took the
  first occurrence's offsets — 32% of exact spans in the ner benchmark
  landed on an already-claimed position; after the port, offset agreement
  with upstream on repeated mentions with matched counts is 90.9%, on par
  with unique mentions. `exact_algorithm: :first_occurrence` restores the
  old behavior.
- **Chunk stream no longer blocks on the slowest chunk** — the orchestrator's
  `Task.async_stream` now runs `ordered: false` (document order is restored
  by sorting chunk results on their byte offsets), so one slow chunk — e.g. a
  429 riding Req's retry backoff — no longer gates every later chunk launch.
  The default `:max_concurrency` also rises from 3 to 10, matching upstream
  langextract's `max_workers` default.
- **Minimum Elixir raised to 1.15** — plug 1.20 (test dependency, pulled in by
  a security patch) requires Elixir 1.15, and CI can no longer verify 1.14.

### Fixed

- **Claude provider default model updated to `claude-sonnet-5`** — the previous
  default, `claude-sonnet-4-20250514`, was retired upstream on 2026-06-15, so
  `LangExtract.new(:claude)` without an explicit `:model` returned 404s.
- **Gemini provider default model updated to `gemini-3.5-flash`** — tracking
  upstream langextract's default (their #472); `gemini-2.0-flash` is
  approaching retirement, the same failure class as the Claude default.
- **Claude provider no longer sends `temperature` by default** — claude-sonnet-5
  rejects non-default sampling parameters with a 400, so the old
  `temperature: 0` default broke every request. It is now sent only when the
  caller explicitly sets `:temperature`.
- **`WireFormat.normalize/1` no longer corrupts YAML block scalars** — the
  colon-quoting pass treated block scalar headers (`dialogue: |-`) as values
  and quoted them, orphaning the indented lines and failing the parse.
  Claude Sonnet 5 emits multi-line extractions as block scalars, so this
  caused chunk-level `{:invalid_format, _}` failures.
- **`WireFormat.normalize/1` parses first, repairs only on failure** — valid
  YAML (including multi-line plain scalars) is never rewritten. The repair
  pass now also recovers unterminated and mis-escaped quoted values and folds
  plain-scalar continuation lines, fixing all chunk failures observed in the
  July 2026 benchmark (11/11 payloads, 90 extractions recovered).

## [0.4.0] - 2026-07-02

### Changed

- **`LangExtract.IO` renamed to `LangExtract.Serializer`** (breaking) — The old
  name shadowed Elixir's standard-library `IO` module, forcing callers to alias
  around the collision. The functions are unchanged.
- **`LangExtract.Pipeline.Extraction` promoted to `LangExtract.Extraction`**
  (breaking) — The struct users build in every template example is the
  library's central payload, shared by `Prompt` and `Pipeline` alike; it now
  lives at the top level instead of inside one consumer's namespace.
- **`LangExtract.Pipeline.FormatHandler` renamed to `LangExtract.WireFormat`**
  (breaking) — The LLM wire-format port (encode for prompts, decode for
  responses) moved to the top level for the same reason. With both moves,
  `Prompt` no longer depends on `Pipeline` at all.
- **Provider HTTP defaults: 120s receive timeout and transient retries** —
  Reverses the 0.2.0 "retries disabled by default" decision. LLM completions
  routinely exceed Req's 15s `receive_timeout` default, and `retry: false`
  meant a transient 429/5xx permanently dropped a chunk as a `ChunkError`.
  Both remain overridable via `req_options:`.
- **Aligner ports upstream langextract v1.6.0 semantics** — After the exact
  phase, a difflib-style lesser phase grounds partial matches anchored at the
  extraction's first token, and an LCS subsequence fallback (with upstream's
  0.75 coverage and 1/3 density gates, plus light plural stemming) replaces
  the fixed-window fuzzy matcher. Extractions that previously returned
  `:not_found` (interrupted dialogue, plural variants) now ground as `:fuzzy`
  with trimmed spans. New options: `:min_density`, `:accept_lesser`.
  Verified against upstream via generated differential fixtures
  (`test/fixtures/alignment_parity.json`).
- **Exact alignment via linear scan** — Replaces `List.myers_difference/2`,
  which did O(N²) work in source token count and missed genuinely contiguous
  matches when extraction tokens also appeared scattered earlier in the source
  (those fell back to `:fuzzy`; they now align as `:exact` with the same byte
  offsets).
- **Hex package no longer ships the benchmark Mix task** — the
  `benchmark.run` task needs the local `benchmark/` corpus, which was never
  packaged, so the task
  could only fail for downstream users. An explicit `files:` list now scopes
  the package to the library itself.

### Added

- **Verbatim extraction instruction in prompts** — `Prompt.Builder` now
  instructs the model to extract only verbatim spans and to emit
  `extractions: []` for contentless passages. Reduces ungrounded extractions
  (few-shot echoes, merged interrupted quotes) that could never align.
- **`Serializer.span_to_map/1`** — Public single-span serialization
  (previously private), also used by the benchmark task instead of a
  duplicated implementation.

### Fixed

- **`Serializer.from_map/1` and `load_jsonl/1` no longer raise on malformed
  input** — An unknown or missing extraction `"status"` now returns
  `{:error, :invalid_data}` (the module's existing error contract) instead of
  raising `ArgumentError` from `String.to_existing_atom/1`.

## [0.3.0] - 2026-04-06

### Changed

- **`LangExtract.run/4` returns `{:ok, {spans, chunk_errors}} | {:error, reason}`** —
  Always returns partial results alongside chunk errors instead of halting on
  the first failure. Infrastructure failures (task exits, timeouts) return
  `{:error, reason}`.
- **Pipeline namespace** — `FormatHandler`, `Parser`, `Extraction` moved under
  `LangExtract.Pipeline.*`. `Pipeline` is the public API for the extraction context.
- **YAML format with quoting** — LLM wire format switched from JSON to YAML,
  matching the upstream Python library. Unquoted values containing colons are
  automatically quoted before parsing.
- **Removed `run_single`** — All text goes through chunking, matching Python's
  behavior. The `max_chunk_chars: :disabled` option is removed.
- **Removed `:on_chunk_error` callback** — Errors are now visible in the return
  value. The callback was redundant.
- **Removed previous chunk context** — Was causing cross-chunk `not_found`
  alignments. Python disables this by default.
- **`Chunk` struct** now includes `byte_end`, computed once in `pack_sentences`.
- **`FormatHandler.normalize/1`** passes through valid YAML without an
  `extractions` key, letting `Parser` return `:missing_extractions`.
- **Broke dependency cycle** between `LangExtract` and `Orchestrator`. Shared
  pipeline logic (normalize → parse → align) extracted into `LangExtract.Pipeline`.
- **Reuse Req HTTP client** across requests. New `build_http_client/1` callback
  on `Provider` behaviour builds the `Req` struct once at `new/2` time, stored
  on `Client.http_client` and reused for all subsequent requests.
- **Tokenizer `classify/1`** uses binary pattern matching for ASCII bytes,
  falling back to Unicode regex only for non-ASCII. Avoids up to 3 regex calls
  per token.
- **`Client` struct** now redacts `:options` and `:http_client` from `inspect`
  output to prevent accidental API key exposure in logs.

### Added

- **`LangExtract.Pipeline.ChunkError`** — Struct with `byte_start`, `byte_end`,
  and `reason` for failed chunk regions.
- **Benchmark improvements** — Per-document JSON files in timestamped
  directories, `--document` flag for single-document runs, `_latest` symlink.

## [0.2.2] - 2026-03-19

### Added

- **ROADMAP.md** — Documents future improvements and unported features from
  the original Python library.
- **Aligner edge-case tests** — Additional test coverage inspired by the
  Python langextract test suite.

### Changed

- **README.md** — Moved future improvements to ROADMAP.md. Cleaned up
  comparison section.

### Removed

- **`docs/` directory** — Removed historical design specs and implementation
  plans (17 files, ~7,000 lines). These served their purpose during
  development; the project is now documented via README, CHANGELOG, and ROADMAP.

## [0.2.1] - 2026-03-19

### Fixed

- Remove stale `httpower` entry from `mix.lock`.

## [0.2.0] - 2026-03-19

### Changed

- **Replaced HTTPower with Req** as the HTTP client. Req is a mature,
  batteries-included HTTP client with wide ecosystem adoption. This removes
  the `httpower` and direct `finch` dependencies.
- **Gemini API key** now passed via Req's `params:` option instead of being
  embedded in the URL path string.
- **Req retries disabled by default** in all providers. Callers can opt in
  via `req_options: [retry: :transient]`.
- **Generic `:req_options` passthrough** replaces the test-specific `:plug`
  option. Any Req configuration (timeouts, retry, pool settings, plug for
  testing) can be forwarded to the underlying Req request.

### Added

- **Orchestrator with chunking** — `LangExtract.run/3,4` wires the full
  pipeline end-to-end. Sentence-aware chunking via `:max_chunk_chars` option
  with `Task.async_stream` for parallel inference.
- **`LangExtract.new/2`** — Req-inspired two-step API: create a client, then
  run extractions.
- **`LangExtract.Chunker`** — Sentence-aware text splitting with
  abbreviation awareness and three-tier strategy.
- **`LangExtract.IO`** — Serialize extraction results to plain maps and JSONL.
- **Module reorganization** — Alignment and Prompt subdomains for cleaner
  namespace organization.

## [0.1.0] - 2026-03-18

Initial release. A complete Elixir port of the core pipeline from
[google/langextract](https://github.com/google/langextract) — extracts
structured data from text using LLMs and maps every extraction back to exact
byte positions in the source.

### Added

#### Core Pipeline

- **`LangExtract.new/2`** — Create a configured LLM client with a provider
  shorthand (`:claude`, `:openai`, `:gemini`) and provider-specific options.
- **`LangExtract.run/3,4`** — Run the full extraction pipeline: build prompt →
  call LLM → normalize → parse → align → return enriched spans.
- **`LangExtract.extract/3`** — Parse raw LLM output and align extractions
  against source text. Accepts both canonical (`class`/`text`/`attributes`)
  and dynamic-key format.
- **`LangExtract.align/3`** — Align extraction strings to byte spans in source
  text without LLM involvement.

#### Alignment (`LangExtract.Alignment.*`)

- **Tokenizer** — Regex-based tokenizer producing tokens with byte offsets.
  Keeps contractions as single tokens for better English alignment.
- **Two-phase Aligner** — Phase 1: exact contiguous match via
  `List.myers_difference/2`. Phase 2: fuzzy sliding-window fallback with
  configurable threshold (default 0.75). Uses tuples for O(1) index access.
- **Span struct** — Holds extraction text, byte offsets (`byte_start`,
  `byte_end`), alignment status (`:exact`, `:fuzzy`, `:not_found`), plus
  optional `class` and `attributes` from the LLM.

#### Prompt Building (`LangExtract.Prompt.*`)

- **Template** — Struct holding a task description and few-shot examples.
- **ExampleData** — Struct for a single few-shot example (source text +
  expected extractions).
- **Builder** — Renders Q&A-formatted prompts with dynamic-key extraction
  examples. Supports cross-chunk context via `:previous_chunk` option.
- **Validator** — Pre-flight check that few-shot examples align against their
  own source text. `validate/1` returns results; `validate!/1` raises.
  The caller decides severity — no built-in logging or severity levels.

#### Format Handler

- **`LangExtract.Pipeline.FormatHandler`** — Hexagonal port between external LLM format
  and internal domain. Serializes `Extraction` structs to dynamic-key JSON for
  prompts. Normalizes raw LLM output (strips `<think>` tags, markdown fences,
  converts dynamic keys to canonical `class`/`text`/`attributes` format).
  Returns decoded maps to avoid redundant JSON round-trips.

#### LLM Providers

- **Provider behaviour** — Single `infer/2` callback. Shared helpers for API key
  resolution (`fetch_api_key/2`), common options (`common_opts/2`), and HTTP
  error mapping (`map_response/2`).
- **Claude** (`LangExtract.Provider.Claude`) — Anthropic Messages API via
  Req. `x-api-key` header auth.
- **OpenAI** (`LangExtract.Provider.OpenAI`) — Chat Completions API via
  Req. Bearer auth. Optional JSON mode (`:json_mode` option, default
  `true`). Works with any OpenAI-compatible endpoint.
- **Gemini** (`LangExtract.Provider.Gemini`) — REST API via Req. Query
  parameter auth. JSON output via `responseMimeType`.

#### Chunking

- **`LangExtract.Chunker`** — Sentence-aware text chunking with three-tier
  strategy: sentence packing → newline splitting → token fallback.
  Abbreviation-aware sentence detection (`Mr.`, `Dr.`, etc.).
  Newline + uppercase heuristic for paragraph breaks.
- **Orchestrator chunking** — When `:max_chunk_chars` is set, the orchestrator
  splits the source, processes chunks in parallel via `Task.async_stream`,
  adjusts byte offsets, and concatenates results. Previous chunk text is passed
  as prompt context for cross-chunk coreference resolution.

#### I/O

- **`LangExtract.IO`** — Serialize extraction results to plain maps
  (`to_map/2`) and back (`from_map/1`). Save/load multiple results as JSONL
  (`save_jsonl/2`, `load_jsonl/1`).

#### Infrastructure

- **Client struct** — Holds provider module and options. Created via
  `LangExtract.new/2`.
- **Req** — Batteries-included HTTP client. Uses `json:` option for automatic
  request body encoding. Retries disabled by default; opt in via `:req_options`.
- **Req.Test** — All provider integration tests use stubs, not network
  calls.
- **Credo** — Strict mode passes with zero issues.
- **187 tests** — Full coverage across all modules.

### Divergences from Python Reference

- **Byte offsets** instead of character offsets (natural for Elixir binaries).
- **Contraction handling** — `don't` is one token, not three.
- **No `MATCH_LESSER`/`MATCH_GREATER`** — Deliberate simplification. Our
  three statuses (`:exact`, `:fuzzy`, `:not_found`) are cleaner.
- **Claude provider** — Not in the original; added as the primary provider.
- **JSON only** — No YAML support (modern LLMs handle JSON well).
- **Caller-decides severity** for prompt validation (no built-in severity enum).
- **Req-inspired API** — `new/2` + `run/3,4` instead of a single function with
  many keyword arguments.

[0.9.0]: https://github.com/mdepolli/lang_extract/compare/v0.8.0...v0.9.0
[0.8.0]: https://github.com/mdepolli/lang_extract/compare/v0.7.0...v0.8.0
[0.7.0]: https://github.com/mdepolli/lang_extract/compare/v0.6.0...v0.7.0
[0.6.0]: https://github.com/mdepolli/lang_extract/compare/v0.5.0...v0.6.0
[0.5.0]: https://github.com/mdepolli/lang_extract/compare/v0.4.0...v0.5.0
[0.4.0]: https://github.com/mdepolli/lang_extract/compare/v0.3.0...v0.4.0
[0.3.0]: https://github.com/mdepolli/lang_extract/compare/v0.2.2...v0.3.0
[0.2.2]: https://github.com/mdepolli/lang_extract/compare/v0.2.1...v0.2.2
[0.2.1]: https://github.com/mdepolli/lang_extract/compare/v0.2.0...v0.2.1
[0.2.0]: https://github.com/mdepolli/lang_extract/compare/v0.1.0...v0.2.0
[0.1.0]: https://github.com/mdepolli/lang_extract/releases/tag/v0.1.0