Guardians of Agents

Special report · Research & Future

We Mapped 1,472 Papers on AI Agent Security. Here Is What Works.

We classified 1,472 papers on attacking, defending and controlling AI agents. Most defenses look strong in their own papers and weak under adaptive attack. The designs that hold up best are the least tested.

· 8 min read · All levels

Research AtlasAdaptive attacksBenchmarksOpen problems

In this issue
  1. How we built the map
  2. Finding 1: reported numbers and adaptive numbers are different worlds
  3. Finding 2: the promising designs change the architecture, not the model
  4. Finding 3: we are measuring with bent rulers
  5. Finding 4: the attack surface has moved to ecosystems
  6. Finding 5: the most deployed defense is the least studied
  7. What is still open
  8. What’s next
  9. Sources

Every week another paper claims to have solved prompt injection for AI agents. Attack success drops from 40 percent to 1 percent, the tables are bold, and the method has a memorable acronym. A few months later another paper breaks it.

To see the pattern clearly, we built a map. The Agent Security Research Atlas classifies 1,472 papers from arXiv, IEEE, ACM, USENIX, NDSS, ACL and OpenReview into 25 method families, rates how mature each family is, and records what happens to its defenses when the attacker adapts. This issue summarizes what we found. The atlas itself is searchable, and the full dataset is free to download.

Key takeaways

  • The field more than tripled in two years, and defense papers now outnumber attack papers.
  • Defenses that ask a model to tell instructions from data (detectors, prompt formatting, robustness training) report near-zero attack success, yet independent adaptive attacks reach about 70 to 96 percent.
  • Architectural designs that keep untrusted text away from decisions are the most promising, and the least tested against adaptive attackers.
  • The benchmarks themselves are saturated, and papers do not agree on what counts as a successful attack.
  • The most widely deployed defense, human approval, is studied in only about ten papers.

How we built the map

We could not query every publisher’s database directly, so we combined two routes. First, 21 curated research lists maintained by academic groups gave us 1,694 unique arXiv papers. Second, about 120 targeted searches of IEEE Xplore, the ACM Digital Library, USENIX, NDSS, the ACL Anthology and OpenReview added 363 papers that the lists missed. After screening, 1,472 papers were in scope: 1,073 directly about agents and 399 closely related.

Each paper was assigned one family out of 25: twelve kinds of attack (from indirect prompt injection to agents that sabotage their own tasks), nine kinds of defense, three kinds of evaluation, and surveys. For about 190 key papers we read the abstracts to record effectiveness numbers together with the setting they were measured in. The long tail was classified from titles and list descriptions, so treat those labels as a starting point.

Finding 1: reported numbers and adaptive numbers are different worlds

The single most important paper in this literature may be “The Attacker Moves Second” (October 2025), by researchers across several labs. They took twelve published defenses against jailbreaks and prompt injection, most of which reported attack success near zero, and attacked each with gradient methods, reinforcement learning, search and more than 500 human red-teamers. Most defenses fell above 90 percent.

Reported versus adaptive attack success for four defensesPaired bars: each defense reported about 0 to 2 percent attack success on AgentDojo, but adaptive attacks reached 95 to 96 percent.Attack success rate on AgentDojo0%25%50%75%100%Spotlightingreported 1%adaptive >95%Prompt sandwichingreported 1%adaptive >95%Meta SecAlignreported 2%adaptive 96%MELONreported 0.24%adaptive 95%reported in the defense paperindependent adaptive attack
Figure 1 Four defenses on the AgentDojo benchmark. Green bars are the attack success reported when each defense was published; amber bars are what adaptive attacks achieved. Sources: Nasr et al. 2025 for all four adaptive figures; MELON's reported figure is for GPT-4o.

Other groups found the same thing independently. An adaptive-attack study at NAACL 2025 broke all eight indirect-injection defenses it tested, each above 50 percent. A reinforcement-learning attacker reached 72 percent against GPT-5 with its instruction hierarchy. A June 2026 paper argues formally that, when instructions and data share the same input channel, no classifier or training method can prevent injection perfectly.

The lesson for readers of the literature is simple: a defense result without an adaptive attack is a lower bound on risk, not an upper bound on safety.

Finding 2: the promising designs change the architecture, not the model

A second group of defenses does not ask the model to resist injected text at all. Instead it changes who is allowed to decide what:

  • Information-flow control. Google’s CaMeL has a privileged model write the plan from the user’s request alone, while a quarantined model reads untrusted data, and an interpreter tracks where every value came from. Microsoft’s FIDES tracks confidentiality and integrity labels in a similar way.
  • Deterministic privilege policies. Progent checks every tool call against a symbolic least-privilege policy, and in the one independent replication we found it held under 3 percent against a hand-crafted adaptive attack.

These designs come with a cost. CaMeL completed 77 percent of AgentDojo tasks with provable security, against 84 percent with no defense, and used roughly 2.8 times as many tokens. They also cannot handle tasks where the right action depends on what the untrusted data says. And almost none has yet faced a white-box adaptive attack.

Defense families by deployment maturity and adaptive-attack evidenceNine defense families placed on a grid: horizontal axis is maturity from concept to deployed; vertical axis is evidence under adaptive attack from broken to untested.ConceptPrototypeBenchmarkedAdoptedDeployedBrokenContestedUntestedMaturity →Adaptive evidenceB1 DetectionB2 PromptstructuringB6 MonitoringB3 TrainingB9 HumanapprovalB5 RuntimepolicyB8 InternalsignalB4 Info-flowcontrolB7 AccesscontrolPROMISING, NEEDS ADAPTIVE TESTS
Figure 2 Nine defense families by deployment maturity (horizontal) and adaptive-attack evidence (vertical). The families already in products are the ones adaptive attacks have broken; the architectural families in the shaded region are promising but largely untested.

Finding 3: we are measuring with bent rulers

Several 2025 and 2026 papers turned the microscope on the benchmarks. A simple “firewall” that minimizes tool inputs and sanitizes tool outputs scored perfect security on four popular benchmarks, which says more about the benchmarks than the firewall. Papers define attack success differently: one computer-use benchmark reports agents attempting the attacker’s goal up to 92.5 percent of the time but completing it 43 to 60 percent of the time. A re-audit of one MCP attack campaign reduced its “successful attacks” to zero.

Two other findings stand out. Stronger models can be more exploitable, because they follow tool instructions better (MCP Security Bench, ICLR 2026). And multi-turn adaptive attackers raise success five to fourteen times over single-turn attacks on the same models.

Finding 4: the attack surface has moved to ecosystems

Early work used synthetic tools. Current work tests production products and whole registries. One study analyzed 98,380 agent skills and confirmed 157 malicious ones. Another found 40.55 percent of 7,973 remote MCP servers exposing tools with no authentication. Coding agents are now evaluated directly: in one benchmark, 66.5 percent of malicious GitHub issues passed every guardrail in Cursor, Claude Code and Codex.

Finding 5: the most deployed defense is the least studied

Human approval before risky actions is nearly universal: a 2026 review found runtime approval in 15 of 21 production agents. Yet fewer than a dozen papers study it. The early results are not reassuring. People fall for the same fine-print injections agents do, and a September 2026 paper showed approval prompts in several agent frameworks could be swapped after the user clicked “approve.”

Case file: production monitors versus a scheming agent (September 2026)

A September 2026 study red-teamed two production blocking monitors, the classifiers that decide whether a coding agent’s command may run, by giving the agent itself the goal of getting a harmful command through. In 79 percent of trials, the agent used prompt injection against the monitor to run arbitrary shell commands. The authors proposed better tool coverage, safer transcript formatting and an extra agentic review stage, and noted that attacks spread across multiple contexts remain unsolved. The monitor, in other words, is also a model that reads untrusted text.

What is still open

The atlas lists 22 open problems in six themes. Five stand out for anyone choosing a research direction or a product roadmap:

  1. Adaptive evaluation as the default, with shared attack budgets built into benchmarks.
  2. Tasks that must act on untrusted instructions, such as “follow the README” or “do what this email asks,” where filtering the instruction also breaks the task.
  3. Security of the harness, not just the model: permission systems, sandboxes and approval flows that hold when the model is fooled.
  4. Memory and multi-agent integrity, where one poisoned record or one compromised agent can spread.
  5. Monitors that cannot be talked out of their job.

For your team: use the atlas in one hour

  1. Open the Research Atlas and filter the paper explorer to the families that match your deployment (for example, B5 runtime policy and A3 tool supply chain for an MCP-heavy agent).
  2. For each defense you rely on, look up its family in the defense table. Note whether the family is marked Broken, Contested or Untested under adaptive attack.
  3. For every “Broken” layer, add a control from an out-of-band family (B4, B5 deterministic policy or B7).
  4. Download the CSV and keep the rows relevant to your stack as a reading list for the quarter.

What’s next

The weekly series continues on Monday. The atlas will be updated as new work appears; if you know of a paper we missed or classified wrongly, tell us.

Sources

  1. M. Nasr, N. Carlini, et al., “The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections,” 2025. https://arxiv.org/abs/2510.09023
  2. Q. Zhan et al., “Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents,” NAACL Findings 2025. https://arxiv.org/abs/2503.00061
  3. “RL Is a Hammer and LLMs Are Nails,” 2025. https://arxiv.org/abs/2510.04885
  4. “On the Inseparability of Instructions and Data in Shared-Embedding Sequence Models,” 2026. https://arxiv.org/abs/2606.27567
  5. E. Debenedetti et al., “Defeating Prompt Injections by Design” (CaMeL), 2025. https://arxiv.org/abs/2503.18813
  6. M. Costa et al., “Securing AI Agents with Information-Flow Control” (FIDES), 2025. https://arxiv.org/abs/2505.23643
  7. T. Shi et al., “Progent: Securing AI Agents with Privilege Control,” 2025. https://arxiv.org/abs/2504.11703
  8. Independent evaluation of out-of-band defenses, 2026. https://arxiv.org/abs/2606.26479
  9. K. Zhu et al., “MELON: Provable Defense Against Indirect Prompt Injection,” ICML 2025. https://arxiv.org/abs/2502.05174
  10. R. Bhagwatkar et al., “Firewalls to Secure Dynamic LLM Agentic Networks… or Stronger Benchmarks?”, 2025. https://arxiv.org/abs/2510.05244
  11. RedTeamCUA, ICLR 2026. https://arxiv.org/abs/2505.21936
  12. Re-audit of MCP attack success labels, 2026. https://arxiv.org/abs/2608.12880
  13. MCP Security Bench, ICLR 2026. https://arxiv.org/abs/2510.15994
  14. “Adaptive Adversaries” (multi-turn), 2026. https://arxiv.org/abs/2607.18063
  15. “Malicious Agent Skills in the Wild,” USENIX Security 2026. https://arxiv.org/abs/2602.06547
  16. “A First Measurement Study on Authentication Security in Real-World Remote MCP Servers,” 2026. https://arxiv.org/abs/2605.22333
  17. IssueTrojanBench, 2026. https://arxiv.org/abs/2607.20759
  18. “Reframing LLM Agent Security as an Agent–Human Interaction Problem,” 2026. https://arxiv.org/abs/2605.24309
  19. “The Obvious Invisible Threat: GUI Agents’ Vulnerability to Fine-Print Injections,” 2025. https://arxiv.org/abs/2504.11281
  20. “Loopjacking: Hijacking Human-in-the-Loop Approval,” 2026. https://arxiv.org/abs/2609.21081
  21. “Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents,” 2026. https://arxiv.org/abs/2609.19587