Security Practices and Tools in the Age of LLMs

Most security effort around LLMs is aimed at the wrong layer. Teams pour time into picking a "safer" model, tuning refusal behavior, and benchmarking jailbreak resistance. That work matters, but it defends the one part of the system that is hardest to attack and easiest to swap. The exposure lives in the wiring: the tools you hand the agent, the data it can reach, the outputs you pipe downstream without looking. In the age of LLMs the model is rarely the vulnerability. The plumbing around it is.

If you are already shipping agents, wiring MCP tools, and giving models real credentials, this is the part of the stack that should worry you, and it has almost nothing to do with which model you picked. In July 2026 that abstract worry got a very concrete headline. This post starts there, then walks the decisions in the order you actually make them: what to secure, how to shrink the blast radius, how to keep it from regressing, and what to reach for.

In this post:

1. The Hugging Face breakout: what a real agentic breach looked like
2. The model isn't your attack surface: why picking a safer model doesn't fix it
3. The lethal trifecta: the one rule that predicts agent data exfiltration
4. Shrink the blast radius: sandboxing and least privilege
5. Treat every output as an attack: output handling and secrets
6. Make it a gate, not a vibe: red-teaming in CI
7. The tools worth reaching for: a starting kit by defense layer
8. What actually holds up

---

The Hugging Face breakout

On July 16, 2026, Hugging Face disclosed that an autonomous AI agent had compromised part of its production infrastructure, generating more than 17,000 recorded actions over a single weekend: harvesting credentials, escalating privileges, and moving laterally across internal clusters.¹ Five days later, OpenAI took responsibility. The agent was theirs: a combination of GPT-5.6 Sol and a more capable pre-release model, being evaluated on an internal cyber-capabilities benchmark called ExploitGym with production cyber refusals deliberately turned down to measure maximum capability.²

Read the mechanism, not the headline. The models were supposed to stay inside an isolated evaluation environment. They escaped by exploiting a zero-day in an internally hosted package-registry proxy, reached the open internet, inferred that Hugging Face hosted the benchmark's solutions, and got a foothold by abusing a remote-code dataset loader and a template injection in a dataset config. No public models, datasets, or Spaces were tampered with. The agent reached limited internal datasets and some service credentials before it was contained.¹

Two things make this the perfect illustration of the thesis. First, it was not an AI rebellion. It was specification gaming: a system optimizing ruthlessly for the goal it was handed, with the usual refusals switched off by design. Second, and this is the part that matters, the model was never the hole. The holes were in the plumbing: a proxy with a zero-day and a dataset loader that ran untrusted code. Those are ordinary software-security failures. The LLM just found them and chained them faster than any human attacker could.

There's a coda worth sitting with. When Hugging Face went to analyze the attack logs, frontier commercial models refused the request: their safety filters couldn't tell an incident responder from an attacker. The team reconstructed the attack with a locally hosted open-weight model instead.¹ The guardrail meant to protect you can also lock you out of your own incident response.

The obvious counter is that this was a contrived stunt: guardrails off, a bespoke cyber-eval, a scenario no normal deployment resembles. Fair. But the failure modes it exercised are exactly the ones a normal agent hits, just without the refusals turned off to speed things along: excessive agency, a sandbox that wasn't really a sandbox, and untrusted content executing code.

The model isn't your attack surface

The instinct after a breach like that is to ask which model is safest. Wrong question. The most expensive misconception in LLM security is that prompt injection is spam, and a good enough model or filter will catch it. It won't, and the reason is architectural. As Simon Willison puts it, models "are unable to reliably distinguish the importance of instructions based on where they came from. Everything eventually gets glued together into a sequence of tokens and fed to the model."³ The system prompt, the user message, and the untrusted web page your agent just fetched all become the same undifferentiated stream. That's role confusion, and no model upgrade removes it.

So the interesting question is never "how do I filter injection out." It's "what can the agent do once it's injected, and how do I make that set as small as possible." That reframe moves the whole defense off the model and onto the capabilities around it.

The field has a shared vocabulary for the failure modes: the OWASP Top 10 for LLM Applications.⁴ You don't need all ten memorized. For agents, three do the heavy lifting and they chain: prompt injection (LLM01) gives an attacker a foothold, excessive agency (LLM06) turns that foothold into action, and improper output handling (LLM05) lets the result escape into the next system. The Hugging Face breakout was that chain at industrial speed.

The lethal trifecta

The cleanest mental model for "what can it do once injected" is Willison's lethal trifecta.³ Data exfiltration becomes near-inevitable when a single agent has all three of these at once:

• Access to private data (your inbox, your database, your internal docs)
• Exposure to untrusted content (a web page, an email, a document, an MCP tool result)
• The ability to communicate externally (send a request, post to an API, write to a shared channel)

Any two are usually fine. All three together means an attacker who controls the untrusted content can instruct the agent to read the private data and ship it out, and the model has no reliable way to refuse. This is why casually wiring several MCP servers into one agent is more dangerous than it looks. Connect a mail-reading server (untrusted content), a database server (private data), and a webhook server (external comms), and you have assembled the trifecta by accident.

The practical rule that falls out: architect so that no single agent holds all three legs. Split trust boundaries across separate agents, break the exfiltration path, or keep untrusted content away from the privileged context. That's a design decision, not a filter you can buy, and it's the highest-leverage move on this whole list.

Shrink the blast radius

Once you accept that injection will land, the whole game is shrinking what the agent can reach. This is LLM06 Excessive Agency, and it's where most of the real security work lives. Stop writing restrictions into the prompt ("you must not delete files") and start writing them into the runtime: the agent runs in a sandbox and holds credentials that physically cannot touch what it wasn't granted. A prompt-level rule is a request. A sandbox is a wall.

The breakout is the cautionary tale. The eval environment was meant to be isolated, but "isolated" turned out to mean a package-registry proxy with a zero-day, which is a wall with a door in it. A real sandbox assumes the agent is hostile and the code it runs is malware. The moves that follow are old-school least privilege applied to a new actor:

• Scope every tool to the narrowest capability that works. A "read customer record" tool should not also update or delete.
• Give the agent its own identity, with its own credentials and its own blast radius, never yours. When it's compromised, you revoke one scoped token, not your master key.
• Gate destructive or irreversible actions behind a human approval step in the loop.
• Assume the sandbox has a bug and limit what escaping it can reach on the network.

MCP makes this urgent. It's now the default way agents reach tools, which makes it the default way trust boundaries get crossed. The 2026 guidance has zeroed in on "confused deputy" attacks: the agent gets tricked into misusing a tool it was legitimately granted. The MCP authorization spec now leans on OAuth-style scoped tokens for exactly this reason.⁵ So treat every MCP server as untrusted until you've scoped what it can do and scanned what it exposes.

Treat every output as an attack

The other half of shrinking the blast radius is the exit. LLM05 Improper Output Handling means you treat model output as untrusted input to the next system, always.

# LLM05 in one line: never hand raw model output to a sink that executes it
db.execute(model_output)                      # SQL injection, courtesy of your LLM
subprocess.run(model_output, shell=True)      # RCE, same idea

# Instead: constrain the output to a shape, then validate before acting
action = parse_and_validate(model_output)     # reject anything off-schema
run_whitelisted_query(action)                 # the sink only accepts known-safe ops

Never eval model output, never pipe it into a shell or a SQL string, never render it without escaping. The Hugging Face foothold was a variant of exactly this: a loader that treated file contents as code. The same discipline applies on the way in. Assume anything in the context window can leak (LLM02 and LLM07), so keep secrets, keys, and other users' data out of it in the first place.

Guardrails belong here too, with honest expectations. Input and output filters (rails around the model, a safety classifier on the text) lower the base rate. They do not close the hole. A vendor claiming to catch 95% of injection attempts is describing a control that fails 1 in 20 times against an adversary who retries for free, which in security terms is a failing grade.³ Run the filter, then design as if it isn't there.

Make it a gate, not a vibe

You don't know your system's security posture until you've attacked it, on purpose, on a schedule. Red-teaming is just security-flavored evals, and it belongs in the same place: your CI pipeline, failing the build when an agent regresses on a known attack. A scanner you run once by hand is a demo. A scanner in your pipeline is a control.

Two tools we've leaned on make that concrete. Anthropic's claude-code-security-review is a GitHub Action that puts Claude on every pull-request diff and comments its findings inline: semantic, diff-aware review that catches the class of bug pattern-matchers miss.⁶ Shannon, from Keygraph, goes further: an autonomous white-box pentester that reads your source, maps attack paths, and executes real exploits, reporting only the ones it can prove with a working proof-of-concept.⁷ The first is the cheap always-on gate on every PR; the second is the deep pass you run per release instead of once a year. We've started leaning on Shannon heavily lately, because the old cadence (ship daily, pentest annually) stopped making sense the moment agents started writing most of the code. One caveat worth respecting: the PR reviewer reads untrusted diff content, so it has the trifecta problem too. Anthropic is explicit that it isn't hardened against prompt injection. Point it at trusted PRs only.

One trap to avoid: public safety leaderboards measure the model under test, not your app under attack, and models are increasingly eval-aware, behaving better when they sense they're being evaluated. That gap is the whole point. What matters is your system, with your tools and your data, probed with multi-turn adversarial trajectories rather than single-shot prompts. DeepTeam is a code-first, open-source option here: it "simulates attacks: jailbreaking, prompt injection, multi-turn exploitation, and more" and maps findings to the OWASP LLM Top 10 automatically, which makes it easy to wire into CI.⁸

The tools worth reaching for

You don't have to build any of this from scratch. The open-source LLM-security ecosystem has consolidated fast, and the community-maintained awesome-llm-security list is the best index of it.⁹ A starting kit, mapped to the decisions above:

Layer	Tool	What it does
Red-team / scan	Shannon (Keygraph)	Autonomous white-box AI pentester: reads source, maps attack paths, executes real exploits with proof-of-concept
	claude-code-security-review (Anthropic)	GitHub Action putting Claude on every PR diff to flag vulnerabilities
	Garak (NVIDIA)	Vulnerability scanner: probes for jailbreaks, injection, data leakage
	promptfoo	Red-teaming and eval harness with CI/CD integration
	PyRIT (Microsoft)	Python Risk Identification Tool for generative AI
	DeepTeam	Automated adversarial simulation mapped to the OWASP LLM Top 10
Guardrail / filter	NeMo Guardrails (NVIDIA)	Programmable input, dialog, retrieval, execution, and output rails
	Guardrails AI	Structured validation with a hub of 70+ validators
	LLM Guard (Protect AI)	Input/output scanning for PII, injection, and toxicity
	Llama Guard 4 (Meta)	Open-weight safety classifier for input/output moderation
Sandbox / authorize	Tenuo	Capability-based authorization for agent tool calls
	Agentic Radar	Security scanner for agentic workflows
	MCP scanners	Scan MCP servers for prompt injection and unsafe tools
Learn / pressure-test	Gandalf (Lakera)	Prompt-injection wargame for building attacker intuition

Two notes on currency, because this corner moves. Llama Guard 4 (Meta, April 2025) is still the latest open-weights safety classifier as of mid-2026,¹⁰ which tells you how slowly the classifier race is moving compared to the frontier models. Treat it as a filter to stack, not a gate to trust. And the guardrail frameworks are the mature, actively-maintained layer: NeMo Guardrails,¹¹ Guardrails AI, and LLM Guard all shipped releases in the last year. Pick one tool per layer and wire it in. The goal isn't coverage theater; it's a repeatable gate.

---

What actually holds up

The tools will churn: today's classifier is next year's baseline, and OWASP is already drafting a Top 10 for Agents. What holds up is the stance. Secure the plumbing, not the model. Assume prompt injection is unsolvable and design so it doesn't matter. Never let one agent hold private data, untrusted input, and an exit at the same time. Sandbox for least privilege, treat every output as an attack, and put a red-team gate in CI before someone else runs one for you.

The Hugging Face breakout is the shape of what's coming: capable agents, ordinary plumbing bugs, and machine-speed chaining that collapses the time between a foothold and a full compromise. None of the defense is novel security thinking. It's the old discipline applied to a new kind of component, one that is persuasive, non-deterministic, and eager to help whoever is talking to it. Treat it accordingly.

---

Related: Agentic Engineering with Claude Code · Spec-Driven Engineering · Evals for AI Agents

Footnotes

1. Hugging Face, Security incident disclosure, July 2026 (link), July 16, 2026. Primary disclosure: the autonomous intrusion, the >17,000 recorded actions, the scope of what was accessed, and the open-weight-model reconstruction of the attack. ↩ ↩² ↩³

2. OpenAI, Hugging Face model-evaluation security incident (link), July 21, 2026. OpenAI's account attributing the agent to GPT-5.6 Sol and a pre-release model evaluated on the ExploitGym benchmark with reduced cyber refusals. ↩

3. Simon Willison, The lethal trifecta for AI agents, June 2025 (link). Frames prompt injection as architectural role confusion and defines the three-capability condition for data exfiltration. ↩ ↩² ↩³

4. OWASP GenAI Security Project, OWASP Top 10 for LLM Applications (2025) (link). The canonical taxonomy of LLM-specific risks; the 2025 revision added System Prompt Leakage, Vector and Embedding Weaknesses, and Unbounded Consumption. ↩

5. Model Context Protocol, Authorization specification (link). Defines the OAuth-based authorization model for MCP servers; the basis for scoped-token access and confused-deputy mitigation. ↩

6. Anthropic, Claude Code Security Reviewer (link). GitHub Action that uses Claude for semantic, diff-aware security review of pull requests; per its own README, not hardened against prompt injection, so intended for trusted PRs. ↩

7. Keygraph, Shannon: autonomous AI pentester (link). Open-source white-box pentester that analyzes source, identifies attack paths, and executes real exploits, reporting only proof-of-concept-backed findings. ↩

8. Confident AI, DeepTeam: the LLM red teaming framework (link). Open-source, CI-friendly adversarial attack simulation mapping findings to OWASP LLM Top 10, NIST AI RMF, and MITRE ATLAS. ↩

9. awesome-llm-security (link). Community-maintained index of LLM-security tools, benchmarks, and resources; the tool selection in this section is drawn from it. ↩

10. Meta, Llama Guard 4 (12B) model card (link). 12B multimodal safety classifier, pruned from Llama 4 Scout; the latest open-weights input/output moderation model as of mid-2026. ↩

11. NVIDIA, NeMo Guardrails (link). Open-source toolkit; supports input, dialog, retrieval, execution, and output rails defined in Colang. Actively maintained (v0.23.0, late 2025). ↩