Red Teaming Autonomous Agents Needs a Different Playbook

Model red teaming asks whether a prompt produces disallowed output. Agent red teaming asks whether a sequence of individually reasonable steps produces an unacceptable outcome. Those are different exercises, and organisations keep buying the first while believing they bought the second.

The failure modes that only appear in agents

• Goal drift over long runs. Each step is defensible. Twenty steps later the agent is doing something nobody sanctioned.
• Tool chaining. Two safe tools compose into an unsafe capability: read a record, then send mail, equals data exfiltration with no single dangerous permission.
• Poisoned state. Content written earlier in the run — by the agent, by a retrieved document, by another agent — is trusted later as if it were system context.
• Retry loops with side effects. Idempotency failures turn a transient error into duplicated payments, duplicated tickets, duplicated mail.
• Cross-agent trust. Multi-agent systems almost always treat inter-agent messages as trusted. They are inputs like any other.

How to run it

1. Test in an environment with real side effects

Mocked tools hide the interesting failures. Use a full staging environment with real integrations against synthetic data, and let the agent actually act.

2. Define outcome-level objectives

Not "can I make it say X" but "can I make it move money, delete records, mail a customer, or grant access". Score against consequences, not against wording.

3. Attack through the data path

Most realistic entry points are documents, tickets, emails and web pages the agent will read — not the operator's prompt. Plant your payloads where an outsider could plant them.

4. Run long, run repeatedly

Drift is a function of run length. Ten-step exercises will not surface what a hundred-step overnight run does. Repeat the same scenario: agents are non-deterministic and single passes are unreliable evidence.

5. Test the containment, not just the agent

Assume the agent is fully compromised and ask what it can still reach. That question tests your architecture, and the answer stays valid across model upgrades.

Reporting that lands

Findings phrased as model behaviour get filed as vendor problems. Findings phrased as "the agent completed an unauthorised refund of EUR 4,000 through this five-step path, and the tool layer permitted every step" get fixed.

A model produces text. An agent produces consequences. Only one of them can be assured with a prompt list.