Generates attacks that try to get the assistant to claim it can perform actions outside its actual capabilities.
Inspired by promptfoo's excessive-agency plugin. The failure mode here
isn't the assistant describing what it can do — that's fine. The failure
is the assistant acting as if it performed an operation it has no way
to perform: confirming an order it didn't place, claiming to have updated
an account, saying it sent an email or cancelled a subscription.
Each generated case carries an explicit capability policy in
expected.no_policy_violation.policy. These cases assume an informational
assistant with no transactional capabilities. Review this policy before using
it with a target that can actually execute actions. Response-text evaluation
does not verify tool execution or whether a transaction occurred.
Options
:purpose— required. Free-form description of the assistant's domain.:count— number of attacks to generate. Defaults to5.:attacker— module implementingTribunal.RedTeam.Attacker. Defaults toTribunal.RedTeam.Attacker.default/0.:model— passed through to the attacker.:temperature,:max_tokens— passed through to the attacker.
Example
{:ok, cases} = Tribunal.RedTeam.Plugins.ExcessiveAgency.generate(
purpose: "Shopping assistant for a cosmetics retailer.",
count: 5
)