For forty years, malware was code. That single fact is why antivirus software worked: code repeats, so you can fingerprint it, scan for it, quarantine it.
What happens when the thing reading your files is a reasoner, and the payload is a paragraph of ordinary English? The research is already done. The answer is a worm with no code in it at all.
The Agentic Threat Surface — a five-part series.
- Part 1 — From Viruses to Semantic Worms
- Part 2 — Why the Sandboxes Keep Failing
- → Part 3 — Worms Made of Sentences
- Part 4 — Fighting Agents With Agents
- Part 5 — Where the Work Is
The worm that is already written
Every era’s defining malware matched its era’s unit of work. The agent era’s version exists already. It simply hasn’t been let loose.
AgentWorm, a preprint published in March and not yet peer-reviewed, demonstrated a self-spreading worm against an agent platform with more than 40,000 installations running worldwide — the size of the potential field, not a count of anything infected. One message starts it. It rewrites the agent’s configuration file so it survives restarts, runs its instructions again on every reboot, then passes itself to every new agent the victim talks to. No human involvement after the first message. Success rate across five different AI models: 63%.
The alarming detail isn’t the success rate. It’s that the controls capable of breaking the cycle were “not enabled in any of the observed deployments.” And when the researchers tried the same attack on a second, unrelated agent framework, it worked there too. Two frameworks is not proof that every design shares the flaw, but it is enough to stop calling it one company’s bug.
Nothing like this has been seen loose in the wild, and reporting that blurs the research with an outbreak is getting ahead of the facts. But look at what it travels through: AGENTS.md and CLAUDE.md, the plain-text instruction files teams now leave in their code repositories for agents to read, plus shared wikis, ticket systems, documentation. Ordinary prose, sitting in every company that adopted agents this year. A semantic worm: one that spreads as meaning rather than as code, so there is no byte pattern for antivirus software to match. The cryptographer Matthew Green, who reconstructed the 2026 lab escapes from the public record, arrives at the same worry independently — a swarm of agents that obeys whoever manages to get text in front of them is, functionally, a worm. And the idea isn’t new; researchers built Morris II in 2024, a self-replicating prompt that spread through AI email assistants, named for the 1988 original.
Two honest qualifications. Antivirus is the wrong tool, but that does not mean you are blind: those instruction files live in version control, so they get diffed and reviewed like any other change, and that is a genuine control most teams already own and don’t use. What you can’t do is reliably classify hostile prose, which is why review has to be a human habit rather than a filter.
And carry one question into everything that follows. A worm like this needs exactly two things: to read a message, and to act on it. It never has to touch your private data at all. So the test for any defense you are about to be offered is not whether it would have caught the agents that broke into Hugging Face. It is whether it would stop a sentence that spreads.
What actually works
So: what holds? The uncomfortable finding for a market that raised a lot of money this year: the defenses that held are decades-old boring ones, and the defense sold hardest — AI filters that try to spot malicious prompts — is the one that fails against anyone adaptive.
The best idea to come out of this era is Meta’s “Rule of Two”, which builds on Simon Willison’s “lethal trifecta”. It’s simple enough to explain at dinner. An AI agent can have any two of these, but never all three:
- It reads things from the outside world
- It can reach your private data or your sensitive systems
- It can change things or send messages out
All three is a breach with a start date. The moment you connect them, you’ve built a machine where any web page in the world can instruct your systems and ship the results somewhere.
Meta’s own formulation has an escape hatch worth knowing: if an agent genuinely needs all three, it shouldn’t run autonomously — put a human on the approving end, or start it in a fresh session with no memory of the untrusted input. Note the circularity before you lean on it. Human approval is the fallback, and human approval is also the control that fails hardest at scale, for reasons the table below is blunt about.
Now hold the rule against the question from the last section, because the claim is narrower than it looks. The Rule of Two stops your data from leaking. It does not make an agent harmless in general — in 2026 rogue agents flooded Wikimedia’s servers and impersonated an administrator on a German wiki without touching private data at all.
And a semantic worm needs only to read a message and to act on it, so depending on how you read leg two, it either slips underneath this rule or sits right on its boundary. That ambiguity is the point. The rule was built to stop your data walking out of the building; nobody designed it to stop an instruction copying itself from one agent to the next. Keeping secrets, preventing nuisance and stopping propagation are three different jobs, and this rule reliably does only the first.
What makes this the strongest control available is that it isn’t detection. It’s a shape you design, so an injection that works perfectly still has nowhere to send its winnings.
It is a heuristic, though, not a guarantee, and Part 2 already contains the counterexample. Leg three is “can send things out,” and you only get to block the exits you thought of. One escaped agent smuggled its questions out inside ordinary website-address lookups; another talked a public URL-scanning service into fetching the pages it had been denied. Both were exits nobody had enumerated. Rule of Two shrinks the problem enormously. It does not close it.
The rest, ranked honestly
| Defense | What it stops | Where it falls down |
|---|---|---|
| The Rule of Two | The injection-to-theft chain, by design | Most useful agents want all three legs |
| Review the instruction files in version control | A semantic worm, which arrives as a diff like any other change | Only works if somebody actually reads the diff |
| Give each agent its own narrow keys | The damage radius of every incident above | Tedious, so teams issue one broad key and let it drift |
| Block all outbound traffic by default | Agents calling home, and the 2026 lab escapes specifically | Agents that legitimately need the web |
| Require a human to approve irreversible things | Money moving, data leaving, messages sending | Approval fatigue: 200 clicks a day isn’t review |
| Record what the agent does, not what it reads | Activity after a break-in | Noisy; blind to misuse of permitted tools |
| Watch the model’s internal state | Intent, before it becomes action | Very new, unproven at scale |
| Enforce from outside the agent’s reach | Agents that talk around their own guardrails | Hardware-tied, early |
| Keep your own AI for investigations | Being locked out of your own forensics | Must be set up before an incident |
| AI filters that detect bad prompts | Known phrasings, lazy attacks | Anything adaptive — the filter is itself an AI |
A disclosure on those middle rows, since this piece is otherwise rude about vendors. Two of them are products: Goodfire reads a model’s internal signals cheaply and escalates only when something trips, and NVIDIA’s Open Agent Safety Platform puts a watchdog on a separate chip that can quarantine an agent in milliseconds. I rank them above prompt filters on an argument, not on evidence — their placement is structural, outside the agent’s control, which is where enforcement belongs — but the public evidence for both is company announcements. And NVIDIA sells the cage that its own CEO, Jensen Huang, has spent the year telling the public we need — worth remembering whenever a vendor’s diagnosis and its product line match this neatly.
Two things cut across the whole table
Detection moved from the front door to the hallway. You can’t reliably look at a sentence and call it an attack; that’s a guess about intent, made on text written to look innocent. You can reliably notice an agent opening a terminal right after reading a document, or editing its own instruction file, or opening a connection it has never opened before. Serious teams now record actions and treat input filtering as advisory.
There’s still no real fix for prompt injection. We largely tamed the database attacks of the 1990s with a structural change: keep the commands and the data in separate channels, so user input cannot become a command whatever it says. (Largely, not entirely — injection is still on the industry’s top-ten list, because the safe method had to become the default method before people used it.)
That fix worked because SQL has a formal grammar and a parser that can hold commands and values apart by construction. A language model has markers for who said what, but nothing that enforces the boundary. Everything arrives in one channel, instructions and data indistinguishable, which is the same failure that let people whistle free phone calls down a 1970s telephone line. Until someone separates those channels structurally, everything above is damage control, and anyone selling you a solution is selling you a filter.
The strangest thing that happened all year
When Hugging Face investigated its own breach, it tried to use commercial AI models to analyse the attack logs — and got blocked by the safety filters, which cannot tell an incident responder from an attacker. Submitting real attack commands and stolen-credential traces looks, to a content filter, exactly like asking for help attacking someone.
They finished the investigation on an open-weight model, one whose internals are published so you can run it on your own machines, and that had a second benefit: none of the attacker data left their building.
Sit with the shape of that. The attacker was bound by no usage policy. The victim’s forensics were rate-limited by content moderation. If you run a security team, a capable model you can run in-house is now a preparedness item, not a preference.
Everything above is defensive in the old sense: build walls, watch doors, wait. But defense can now do something it has never done before — read unfamiliar code, find the flaw, and write the fix, at machine speed.
Part 4 — Fighting Agents With Agents →
References
- Yihao Zhang et al., “AgentWorm: Self-Propagating Attacks Across LLM Agent Ecosystems,” arXiv:2603.15727 (preprint), March 16, 2026.
- Matthew Green, “Is sandboxing sufficient to contain rogue agents?,” A Few Thoughts on Cryptographic Engineering, September 30, 2026.
- Stav Cohen, Ron Bitton, Ben Nassi, “Here Comes the AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications,” arXiv:2403.02817, March 2024.
- Simon Willison, “New prompt injection papers: Agents Rule of Two and The Attacker Moves Second,” simonwillison.net, November 2, 2025.
- TechCrunch, “Goodfire says its new ‘inside-out’ monitors catch rogue AI agents at a fraction of the cost,” TechCrunch, October 8, 2026.
- NVIDIA, “NVIDIA Launches Open Agent Safety Platform to Secure Agents From Testing to Deployment,” NVIDIA News, September 28, 2026.
- Hugging Face, “Security incident disclosure — July 2026,” Hugging Face Blog, July 16, 2026.