You can't tell your board your AI agents are safe if you've never actually attacked them.

Most AI security tooling stops at policy. A guardrail here, a content filter there, a dashboard that says "protected." None of it answers the question a CISO actually has to answer: if a real attacker tried this against our real deployed agents today, what would happen?

A checklist can't answer that. The only way to answer it is with real attacks, run against your real stack, and an honest accounting of what actually stopped them. Mature security programs already do this for infrastructure with three practices: red team, blue team and purple team. AI agents need the same three.

A red team that never goes stale

Traditional red teaming is a snapshot. A consultant spends two weeks, hands you a PDF, and by the time you've read it the threat landscape has moved on. That cadence was already tight for infrastructure. For AI agents, where new attack patterns surface constantly, it doesn't hold.

An attack catalog for agents should be built the opposite way. Every scenario should trace back to a real, published incident, and new ones should be added within days of the incident breaking, not at the next annual engagement.

Each scenario should model a real technique:

• Prompt injection
• Tool abuse
• Memory poisoning
• Agent impersonation
• Credential stuffing
• Supply chain compromise

Mapping each one to the OWASP Top 10 for LLM Applications and MITRE ATLAS keeps the coverage legible to auditors and to the rest of the security team. When a new attack pattern shows up in the wild, it belongs in the catalog, not in a slide deck.

A blue team that gets tested honestly

Running attacks is the easier half of the problem. The harder half is telling the truth about what happened. A scan that reports "blocked" without saying what blocked it is not much better than no scan at all.

My team found and fixed exactly this gap in our own scan engine. For a class of scenarios where no dedicated control existed yet, a generic filter would occasionally catch the attack language by coincidence. The reporting made that look identical to a purpose-built defense working as intended. The judge that confirms verdicts had no way to say "this wasn't a purpose-built defense, it was incidental."

We fixed the engine to attribute honestly. When there is no dedicated control, the report says so, instead of implying coverage that doesn't exist yet.

That honesty is the point. A red team exercise is only useful if the blue team side of the report is trustworthy. A scan has to distinguish a real, named control blocking an attack from a coincidental catch, so that when the report says "covered," it means something. It's the standard your SOC should be able to hold every vendor's reporting to.

Purple team: the loop that closes gaps

Red and blue teaming in isolation produce a report. Purple teaming produces a fix. The program should be built around that loop, not around the report:

1. Attack. A real scenario runs against your real deployed stack.
2. Attribute. The result is traced to a specific control, or honestly marked as uncovered.
3. Fix. Uncovered gaps get triaged into real engineering work, not a backlog that never moves.
4. Re-validate. The next scan confirms the fix closed the gap, live, not on paper.

We have run this loop dozens of times against our own platform. It found real gaps: a rate-limiting bypass on repeated credential attempts, a stale compliance report that had silently regressed, and the attribution bug in the scan engine itself. Each one was closed with a real fix and re-verified with a real re-scan, not assumed fixed because the code looked right.

What this means for a SOC or CISO team

This gives you the same three things a mature security program already runs for its infrastructure, applied to AI agents specifically:

• A live attack catalog that doesn't go stale.
• Honest attribution you can put in front of an auditor.
• A loop that turns findings into fixes instead of a PDF that sits in a drive.

What to ask before you trust an AI red team report

Whoever is testing your AI agents, ask these before you trust the results:

1. Does the catalog update, or is it a fixed snapshot from onboarding? Attack patterns move fast. Your test coverage should too.
2. Can it run on-prem or air-gapped? If your agents are in a sensitive environment, the tester needs to be able to run there too.
3. Does "blocked" name a real control, or just report a status code? A vague "blocked" can hide a coincidental catch instead of a real defense.
4. What happens after a gap is found? If the answer is "it goes in a report," ask what closes the loop.

The bottom line

Policy tells you what your agents are supposed to do. Only an attack tells you what they actually do. Red team it, blue team it, and then run the loop until every finding is a fix you have re-verified.