LemonRouter.RoutingFeedbackReport (lemon_router v0.1.0)

Offline evaluation and reporting layer for routing feedback.

Provides filtering, confidence annotation, and formatting of routing feedback data. Keeps offline eval logic separate from online routing.

Confidence levels

Confidence is assigned based on sample size and success rate:

Level	Criteria
`:insufficient`	`total < min_sample_size`
`:low`	`total >= min_sample_size` and `success_rate < 0.5`
`:medium`	`success_rate >= 0.5` and `< 0.8`
`:high`	`success_rate >= 0.8`

Minimum sample size

The min_sample_size threshold is read from RoutingFeedbackStore.min_sample_size/0 (default 5, configurable via application config). Confidence is :insufficient until this threshold is met.

Recency

Each entry is annotated with last_seen_ms. Pass since_ms: to filter entries whose most-recent sample predates the cutoff.

Usage

# All fingerprints with confidence annotation
{:ok, entries} = RoutingFeedbackReport.list_all()

# Filter by workspace
{:ok, entries} = RoutingFeedbackReport.by_workspace("/my/project")

# Filter by task family (last 7 days only)
since = System.system_time(:millisecond) - 7 * 24 * 3600_000
{:ok, entries} = RoutingFeedbackReport.by_family(:code, since_ms: since)

# Human-readable output
IO.puts(RoutingFeedbackReport.format(entries))

Summary

Functions

by_family(family, opts \\ [])

Filter entries by task family.

by_workspace(workspace, opts \\ [])

Filter entries by workspace key.

format(entries)

Format a list of annotated fingerprint entries for human-readable output.

list_all(opts \\ [])

List all fingerprints with confidence annotations.

parse_key(key)

Parse a fingerprint key string into a map of components.

Functions

by_family(family, opts \\ [])

@spec by_family(
  String.t() | atom(),
  keyword()
) :: {:ok, [map()]} | {:error, term()}

Filter entries by task family.

The family is the first |-delimited segment in the fingerprint key. Accepts an atom (:code) or string ("code").

by_workspace(workspace, opts \\ [])

@spec by_workspace(
  String.t(),
  keyword()
) :: {:ok, [map()]} | {:error, term()}

Filter entries by workspace key.

The workspace is the third |-delimited segment in the fingerprint key. Pass "-" to match runs with no workspace.

format(entries)

@spec format([map()]) :: String.t()

Format a list of annotated fingerprint entries for human-readable output.

Returns a multi-line string suitable for Mix.shell().info/1.

list_all(opts \\ [])

@spec list_all(keyword()) :: {:ok, [map()]} | {:error, term()}

List all fingerprints with confidence annotations.

Options

:since_ms — exclude entries whose last_seen_ms is older than this Unix millisecond timestamp.

parse_key(key)

@spec parse_key(String.t()) :: map()

Parse a fingerprint key string into a map of components.

Key format: family|toolset|workspace|provider|model

Each segment that equals "-" is returned as nil.

Examples

iex> RoutingFeedbackReport.parse_key("code|bash|/my/proj|anthropic|opus")
%{family: "code", toolset: "bash", workspace: "/my/proj", provider: "anthropic", model: "opus"}

iex> RoutingFeedbackReport.parse_key("query|-|-|-|-")
%{family: "query", toolset: nil, workspace: nil, provider: nil, model: nil}