Tribunal.RedTeam.Plugin behaviour (Tribunal v1.4.0)

Behaviour for red-team attack generators.

A plugin produces a list of adversarial test cases targeted at a specific failure mode (policy violations, prompt extraction, hijacking, etc.). Each generated case is a regular Tribunal dataset entry with metadata identifying the plugin and an expected clause naming the assertion to run against the target's response.

Implementing a plugin

defmodule MyApp.RedTeam.Plugins.Custom do
  @behaviour Tribunal.RedTeam.Plugin

  @impl true
  def id, do: :custom

  @impl true
  def severity, do: :medium

  @impl true
  def generate(opts) do
    # ...
    {:ok, [%{input: ..., metadata: ..., expected: ...}, ...]}
  end
end

Built-in plugins live under Tribunal.RedTeam.Plugins.*. Custom plugins are registered via:

config :tribunal, :red_team_plugins, [MyApp.RedTeam.Plugins.Custom]

Summary

Types

case_t()

Callbacks

generate(opts)

id()

severity()

Functions

all_ids()

Returns the list of registered plugin ids.

all_plugins()

Returns all plugin modules (built-in + custom).

builtin_plugins()

Returns built-in plugin modules.

custom_plugins()

Returns custom plugin modules registered via application config.

extract_attacks(other)

Extracts and validates the attacker's attacks list.

fetch_required(opts, keys)

Fetches required options, returning a friendly error instead of raising.

find(id)

Finds a plugin module by its id/0 value.

Types

case_t()

@type case_t() :: %{input: String.t(), metadata: map(), expected: keyword() | map()}

Callbacks

generate(opts)

@callback generate(opts :: keyword()) :: {:ok, [case_t()]} | {:error, term()}

id()

@callback id() :: atom()

severity()

@callback severity() :: :low | :medium | :high

Functions

all_ids()

Returns the list of registered plugin ids.

all_plugins()

Returns all plugin modules (built-in + custom).

builtin_plugins()

Returns built-in plugin modules.

custom_plugins()

Returns custom plugin modules registered via application config.

extract_attacks(other)

Extracts and validates the attacker's attacks list.

Accepts the attacker response with an attacks list keyed by either string or atom. Every attack must carry a non-empty prompt; otherwise the whole batch is rejected with {:error, {:invalid_attack, attack}} rather than emitting a case with input: nil. An unrecognised shape returns {:error, {:unexpected_attacker_response, other}}.

fetch_required(opts, keys)

Fetches required options, returning a friendly error instead of raising.

Returns {:ok, values} with values in the same order as keys, or {:error, {:missing_options, missing}} listing every absent key. Plugins use this in generate/1 so a missing required option surfaces as a {:error, _} the caller can handle, rather than a raw KeyError.

find(id)

Finds a plugin module by its id/0 value.

Returns {:ok, module} or :error.