LemonRouter.RolloutGate (lemon_router v0.1.0)

View Source

Measurable graduation gates for adaptive features.

Defines the quantitative thresholds that must be satisfied before a feature flag can be promoted from opt-in to default-on. The gates are evaluated against metrics snapshots — they do not read the database directly.

Covered features

:routing_feedback (M7-01)

History-aware model selection is safe to enable by default when all of:

  1. Sample size ≥ 50 — enough recorded runs to trust aggregated statistics. Below this threshold the store returns {:insufficient_data, _} and the router falls back to profile defaults.
  2. Success delta ≥ +0.05 — the history-preferred model improves the observed success rate by at least 5 percentage points compared to the session-default model on the same task fingerprint family.
  3. Retry delta ≤ +0.05 — the retry rate has not increased (or has actively decreased) by more than the tolerance.

:skill_synthesis_drafts (M7-02)

Automated skill draft generation is safe to enable by default when all of:

  1. Candidate count ≥ 20 — enough memory documents evaluated to judge pipeline breadth and quality.
  2. Generation rate ≥ 0.6 — at least 60 % of qualified candidates produce a written draft (pipeline is not over-filtering or crashing silently).
  3. False-positive rate ≤ 0.1 — at most 10 % of generated drafts are blocked by the audit engine (content safety is working correctly and not blocking legitimate skill patterns).

Rollback procedure

If a feature is promoted to default-on and unexpected behaviour is observed:

  1. Immediate kill-switch: set the flag to "off" in ~/.lemon/config.toml or via environment variable:

    echo 'routing_feedback = "off"' >> ~/.lemon/config.toml
# or
export LEMON_FEATURE_ROUTING_FEEDBACK=off

  2. Restart the Lemon agent process (or run mix lemon.setup to reload config in a live system).

  3. No data migration required — the routing feedback store and draft store are read-only from the feature gate's perspective. Disabling a feature stops new data from being written but does not delete existing records.

  4. Re-evaluation: after rollback, investigate the feedback store via LemonRouter.RoutingFeedbackReport.list_all/1 or the draft store via mix lemon.skill draft list. Re-run evaluate_routing_feedback/1 or evaluate_synthesis/1 with fresh metrics before re-enabling.

Usage

metrics = %{
  total_samples: 65,
  success_rate: 0.72,
  baseline_success_rate: 0.62,
  retry_rate: 0.09,
  baseline_retry_rate: 0.13
}

case LemonRouter.RolloutGate.evaluate_routing_feedback(metrics) do
  {:ready, computed} ->
    IO.puts("Gate passed — promote routing_feedback to default-on")
    IO.inspect(computed)

  {:not_ready, reasons, computed} ->
    IO.puts("Gate blocked:\n" <> Enum.join(reasons, "\n"))
    IO.inspect(computed)
end

Summary

Types

computed_routing()
computed_synthesis()
gate_result(computed)
routing_metrics()
synthesis_metrics()

Functions

evaluate_routing_feedback(metrics)

Evaluate the rollout gate for the :routing_feedback feature.

evaluate_routing_from_store(store_stats, fingerprints)

Evaluate the :routing_feedback gate from raw store data.

evaluate_synthesis(metrics)

Evaluate the rollout gate for the :skill_synthesis_drafts feature.

evaluate_synthesis_from_run(run_result)

Evaluate the :skill_synthesis_drafts gate from a pipeline run result.

routing_max_retry_delta_abs()

Maximum allowed retry-rate increase (fractional) for :routing_feedback to graduate.

routing_min_samples()

Minimum recorded runs before :routing_feedback can graduate to default-on.

routing_min_success_delta()

Minimum success-rate improvement (fractional) for :routing_feedback to graduate.

synthesis_max_fp_rate()

Maximum fraction of generated drafts that may be blocked by the audit engine.

synthesis_min_candidates()

Minimum candidate documents evaluated before :skill_synthesis_drafts can graduate.

synthesis_min_generation_rate()

Minimum fraction of candidates that must produce a written draft.

Types

computed_routing()

@type computed_routing() :: %{
  total_samples: non_neg_integer(),
  success_rate: float(),
  baseline_success_rate: float(),
  success_delta: float(),
  retry_delta: float()
}

computed_synthesis()

@type computed_synthesis() :: %{
  total_candidates: non_neg_integer(),
  generated: non_neg_integer(),
  blocked_by_audit: non_neg_integer(),
  generation_rate: float(),
  false_positive_rate: float()
}

gate_result(computed)

@type gate_result(computed) ::
  {:ready, computed} | {:not_ready, [String.t()], computed}

routing_metrics()

@type routing_metrics() :: %{
  total_samples: non_neg_integer(),
  success_rate: float(),
  baseline_success_rate: float(),
  retry_rate: float(),
  baseline_retry_rate: float()
}

synthesis_metrics()

@type synthesis_metrics() :: %{
  total_candidates: non_neg_integer(),
  generated: non_neg_integer(),
  blocked_by_audit: non_neg_integer()
}

Functions

evaluate_routing_feedback(metrics)

@spec evaluate_routing_feedback(routing_metrics()) :: gate_result(computed_routing())

Evaluate the rollout gate for the :routing_feedback feature.

Takes a metrics snapshot and checks all three graduation gates.

Parameters

  • metrics — map with keys:
    • :total_samples — total runs recorded in the feedback store
    • :success_rate — observed success rate with history-preferred routing
    • :baseline_success_rate — success rate without history-preferred routing
    • :retry_rate — observed retry rate with history-preferred routing
    • :baseline_retry_rate — retry rate without history-preferred routing

Returns

{:ready, computed}           # all gates pass — safe to promote
{:not_ready, reasons, computed}  # one or more gates failed

evaluate_routing_from_store(store_stats, fingerprints)

@spec evaluate_routing_from_store(map(), [map()]) ::
  {:pass, [String.t()]} | {:fail, [String.t()]}

Evaluate the :routing_feedback gate from raw store data.

Unlike evaluate_routing_feedback/1, this variant accepts the output of RoutingFeedbackStore.store_stats/0 and RoutingFeedbackStore.list_fingerprints/0 directly, using absolute success/failure rate thresholds instead of deltas.

Returns {:pass, notes} or {:fail, failures}.

evaluate_synthesis(metrics)

@spec evaluate_synthesis(synthesis_metrics()) :: gate_result(computed_synthesis())

Evaluate the rollout gate for the :skill_synthesis_drafts feature.

Takes a metrics snapshot and checks all three graduation gates.

Parameters

  • metrics — map with keys:
    • :total_candidates — candidate documents evaluated by the pipeline
    • :generated — drafts successfully written to the draft store
    • :blocked_by_audit — generated drafts blocked by the audit engine

Returns

{:ready, computed}           # all gates pass — safe to promote
{:not_ready, reasons, computed}  # one or more gates failed

evaluate_synthesis_from_run(run_result)

@spec evaluate_synthesis_from_run(map()) ::
  {:pass, [String.t()]} | {:fail, [String.t()]}

Evaluate the :skill_synthesis_drafts gate from a pipeline run result.

Unlike evaluate_synthesis/1, this variant accepts the structured output of LemonSkills.Synthesis.Pipeline.run/3 (with generated as a list of keys and skipped as a list of {key, reason} tuples) rather than pre-counted integers.

Returns {:pass, notes} or {:fail, failures}.

routing_max_retry_delta_abs()

@spec routing_max_retry_delta_abs() :: float()

Maximum allowed retry-rate increase (fractional) for :routing_feedback to graduate.

routing_min_samples()

@spec routing_min_samples() :: pos_integer()

Minimum recorded runs before :routing_feedback can graduate to default-on.

routing_min_success_delta()

@spec routing_min_success_delta() :: float()

Minimum success-rate improvement (fractional) for :routing_feedback to graduate.

synthesis_max_fp_rate()

@spec synthesis_max_fp_rate() :: float()

Maximum fraction of generated drafts that may be blocked by the audit engine.

synthesis_min_candidates()

@spec synthesis_min_candidates() :: pos_integer()

Minimum candidate documents evaluated before :skill_synthesis_drafts can graduate.

synthesis_min_generation_rate()

@spec synthesis_min_generation_rate() :: float()

Minimum fraction of candidates that must produce a written draft.