Legion.EvalGuard.LLM (Legion v0.5.0)

View Source

A Legion.EvalGuard that asks a model whether the code may run.

Define a guard with a policy written in plain language:

defmodule MyApp.CodeReview do
  use Legion.EvalGuard.LLM,
    policy: """
    Deny code that checks out more than once, exports the whole orders
    table, or reads a cart that is not the current conversation's.
    """
end

def config, do: %{eval_guard: MyApp.CodeReview}

Options are :policy and :model (which defaults to Legion's built-in default model, regardless of the model the agent is configured with). The reviewing model sees the policy, the agent's name, the tool modules the code may call, and the code itself - not the conversation.

Without a :policy the guard falls back to default_policy/0, which denies code that goes after the host rather than the task - shelling out, touching the filesystem or the network directly, reaching into other processes or the runtime:

defmodule MyApp.HostGuard do
  use Legion.EvalGuard.LLM
end

The sandbox already blocks the modules that do most of this, so the default earns its keep against what an allowed tool can be talked into.

This is a blocking review: the agent waits for a second model round trip on every eval. Read the latency note in Legion.EvalGuard before reaching for it. If the review request fails, the code is denied, per the same module's rule that a broken guard denies.

Summary

Functions

The policy used when a guard does not give one: deny code that goes after the host rather than the task.

Functions

default_policy()

The policy used when a guard does not give one: deny code that goes after the host rather than the task.

Useful as a starting point for your own - policy: default_policy() <> "...".