Legion.EvalGuard behaviour (Legion v0.5.0)

View Source

Last check before generated code runs.

A guard sees the code an agent wrote and decides whether the sandbox may evaluate it. It runs after the AST checker accepts the code and before the eval process is spawned, so it is the place for policy the sandbox cannot express: "never loop over checkout", "no bulk export of the orders table".

defmodule MyApp.CodeReview do
  @behaviour Legion.EvalGuard

  @impl true
  def check(code, _context) do
    if String.contains?(code, "checkout") and String.contains?(code, "Enum.each") do
      {:deny, "looping over checkout is not allowed - check out once"}
    else
      :allow
    end
  end
end

Guards are off unless configured, globally or per agent:

config :legion, :config, %{eval_guard: MyApp.CodeReview}

def config, do: %{eval_guard: MyApp.CodeReview}

context carries :agent, :agent_id, and :tools (the modules the code may call).

A {:deny, reason} reaches the agent as an execution error, the same path a runtime error takes, so it can rewrite the code or explain itself to the user. Make the reason something an LLM can act on.

Legion.EvalGuard.LLM writes the guard for you from a policy in plain language:

defmodule MyApp.CodeReview do
  use Legion.EvalGuard.LLM, policy: "Deny code that checks out more than once."
end

Guards run on the critical path - the agent waits for every one, so that review costs a model round trip on every eval. Prefer returning :allow and reviewing asynchronously until the async verdicts show blocking is worth it.

A guard that raises, exits, or returns anything other than a verdict is itself a denial - a safety boundary that stops working has to stop the code with it. The [:legion, :eval_guard, :denied] event carries the failure as its reason, so a broken guard shows up as every eval being refused. It is also logged as an error - nothing the agent does can fix it, so it needs a reader who can.

Summary

Types

context()

@type context() :: %{
  agent: module(),
  agent_id: Legion.Store.agent_id(),
  tools: [module()]
}

Callbacks

check(code, context)

@callback check(code :: String.t(), context :: context()) :: :allow | {:deny, String.t()}