Inline @eval annotations for description tuning (spec §4).

Attach evals to a tool to continuously grade the rendered descriptions it advertises across model × verbosity permutations, so terser or model-specific variants (spec §2/§3) do not silently degrade call quality.

Declaring evals

On a classic use Noizu.MCP.Server.Tool module, pass the :evals option — a list of eval specs:

use Noizu.MCP.Server.Tool,
  description: "...",
  evals: [
    [
      name: :simple_task,
      prompt: [%{role: "user", content: "Find recent docs about X."}],
      rubric: [
        covers_pitfall: "resulting call includes the required filters",
        fresh: "mentions post-cutoff knowledge-base entries"
      ]
    ]
  ]

On a use Noizu.MCP.Server.Toolkit function, annotate with @eval (module attribute, accumulate: true) — each @eval drains onto the following @mcp tool, mirroring how @mcp itself is collected:

@eval name: :simple_task, prompt: [...], rubric: [...]
@mcp description: "...", input: [...]
# ⟦𓉧𓀞𓊆𓉥⟧ read_file :: auto-generated pointer for public function read_file
def read_file(args, ctx), do: ...

Each eval spec is a keyword list:

• :name (required) — an atom or non-empty string; unique per tool
• :prompt (required) — a non-empty list of messages or a non-empty string
• :rubric (required) — a non-empty keyword list of criterion: "description" (each description a string)

Malformed specs are a compile error (ArgumentError from the classic tool DSL, CompileError from the toolkit DSL, matching each path's existing convention).

Introspection

list/1 returns [{tool_name, [%Noizu.MCP.Eval.Spec{}]}] for every tool on a server module that carries evals — the entry point for the harness and any app-layer tooling.

Harness

mix noizu.mcp.eval (delegating to Noizu.MCP.Eval.Harness) renders each tool's wire schema through the §0/§2/§3 resolution pipeline for every selected permutation, runs the eval prompt against a target via a pluggable Noizu.MCP.Eval.Runner, and grades each rubric criterion via a pluggable Noizu.MCP.Eval.Judge. Adapters are selected via the :noizu_mcp :eval_runner / :eval_judge application env; a deterministic no-LLM stub pair ships for tests/CI. See those modules for the callback contracts.