Your LLM Passed Its Safety Test. Your Agent Didn't.

The ClawSafety paper ran 2,520 trials across 120 test cases and 5 frontier LLMs. Attack success rates in agent mode hit 40 to 75 percent. Claude Sonnet 4.6 — the safest model in chat-mode benchmarks — still failed 40 percent of agent-level attacks. GPT-5.1 failed 75 percent. LLM agent safety is not a model property. It is a property of the model plus the framework it runs in plus the access the agent has been granted. Enterprise teams building on the assumption that a safe model means safe agents are not being conservative. They are running an empirical gamble with numbers this sharp.

The narrative is undeniably seductive. Pick the safest LLM on the market, build your agent around it, and your safety work is done. The reasoning is straightforward: if the model refuses to generate harmful content in a chat context, it will refuse to take harmful actions in an agent context. The model carries its safety training into every deployment. Buying the right model solves the safety problem.

I think that assumption is empirically false. And the evidence has been sitting in a paper that most enterprise AI teams have not read.

ClawSafety (arXiv:2604.01438) ran 2,520 trials across 120 test cases and 5 frontier LLMs. The headline finding is not subtle: attack success rates in agent mode hit 40 to 75 percent depending on the model. Claude Sonnet 4.6, the safest model in chat-mode benchmarks, still failed 40 percent of agent-level attacks. GPT-5.1 failed 75 percent. The model that won the safety leaderboard in one context became a liability in another.

Here is the fundamental problem with the 'safe model, safe agent' framing. It treats safety as a property of the model. The evidence says it is a property of the model plus the framework it runs in plus the access the agent has been granted. Change any of those three and the safety profile changes.

I want to make a distinction that cuts through the confusion before we go further.

Chat safety is a refusal test. You ask the model to say something harmful directly. The model declines. Pass.

Agent safety is an execution test. An adversarial prompt arrives through an injection channel — email, workspace, web content — and the model has the tools to carry it out. The question is not whether the model would refuse if asked directly. The question is whether it refuses when the prompt arrives sideways, wrapped in legitimate-looking context, with a tool path available.

Those are not the same exam.

The ClawSafety data shows the gap in the starkest possible way. The trust gradient is almost a perfect illustration. Skill and workspace injection produced a 69.4 percent attack success rate. Web content injection produced 38.4 percent. That is not a rounding error. That is nearly a doubling of lethality based on which channel the adversarial prompt travels through. The attack surface grows with the agent's access, and the chat-trained refusal reflexes do not generalise to the agent context.

This distinction — between what a model refuses to say and what it will do when given tools and an injection vector — is the same architectural split that separates LLM reasoning from deterministic execution. Both are about recognising that collapsing two different jobs into one component produces failures that look like model problems but are actually architecture problems.

The model that passed every chat safety benchmark in the world still executed an adversarial instruction 40 percent of the time once it had tools, memory, and an injection vector. That is not a model failure. That is a category error on the part of everyone assuming the two are equivalent.

I am not arguing that the models are broken. I am arguing that the assumption is broken. There is a difference, and the enterprise teams deploying agents at scale need to understand which one they are facing.

The positive case is not hopeless. It is just different from what most teams are doing.

Model selection is not a safety strategy. It is a capability decision with safety implications. You pick the model for what it can do. That decision has safety consequences, and you should know what they are. But you cannot treat the model selection as the safety decision and move on. Framework choice is a co-equal safety decision, and right now most teams treat it as an implementation detail.

The data supports this directly. Scaffold choice shifted attack success rate by 8.6 percentage points. Safety is a model-framework pair property. You cannot isolate one side of that equation and call the work done.

There is a parallel here to the autonomy-versus-authority problem: the agent that deletes your production database is not a model failure either. In both cases, the enterprise is treating a structural decision as if it were a model quality question. The agent had the credential. The agent had the tools. The model did what it was built to do with what it was given. That pattern — conflating what the model is with what the system around it allows — keeps showing up in production incidents.

There are three concrete things I want enterprise teams to do with this.

1. Treat framework choice as a safety decision, not an integration decision.

The 8.6 percentage point swing from scaffold choice means the framework you build your agent on either does real safety work or fails at it. Before you lock in a framework, ask what its injection surface looks like. Does it sandbox tool execution? Does it isolate memory from prompt context? Does it distinguish between user-intent messages and inbound content from external channels? These are not fringe concerns. They are the difference between a 40 percent failure rate and something closer to manageable.

2. Understand which safety boundaries are model-specific and which are not.

Claude Sonnet 4.6 achieved a 0 percent credential-forwarding attack success rate. Every other model in the study failed at least some credential-forwarding attempts. That is a real, hard boundary at the agent level, and it is model-specific, not industry-standard. If your agent handles credentials, this matters. If your framework does not protect credentials and your model is not Claude Sonnet 4.6, you are relying on the model to hold a boundary that the data says it may not hold.

3. Stop treating integration as separate from safety.

The Anthropic integration barrier showed a 46 percent attack success rate. That number lives in the same category as the model failure rates. The tools and connectors that make your agent useful are the same channels that make it vulnerable. The integration work is not a separate track from the safety work. It is the safety work. Every connector you add is an injection surface. Every tool you expose is an execution path. The question is not whether your model is safe. The question is whether your agent architecture treats every integration point as a potential attack vector. The data says it will be treated that way by someone.

At the fleet level, proportional governance — matching controls to blast radius across agent classes — is the structural answer to this problem when you have multiple agents running with different access scopes and autonomy levels. Safety is not a one-agent problem. It is an architecture problem that scales with the number of agents and the surface they touch.

Chat safety tests what the model won't say. Agent safety tests what the model won't do. A model can ace the first and fail the second. Every enterprise deploying agents is treating them as the same exam.