I had this newsletter mostly written before today. It was about the Hugging Face breach they disclosed last Friday, and the angle was the defender-cost asymmetry; their team couldn't get the frontier models to help analyze the attack, because the guardrails wouldn't look at exploit code, so they ran the forensics on open weights instead. It was a good story with a real lesson that lined up nicely with the piece I wrote a couple weeks back.
Then OpenAI published a blog this morning, and I deleted it all and started over.
It turns out that the "attacker" wasn't an attacker. It was a couple of OpenAI's own models, GPT-5.6 Sol and an unreleased one, running an internal cyber-capabilities benchmark with the production guardrails turned off. Nobody pointed the models at Hugging Face. The model was trying to win a test, worked out that Hugging Face probably had the answer key, and broke in to get it.
I've read a lot of incident write-ups. I’venever read one quite like this.
What happened
The exploit chain is worth breaking down because the details are the point. OpenAI runs an eval that pushes a model to chain exploits along hard attack paths, to see how far its cyber capability reaches. To find the ceiling, they turn off the classifiers that would normally stop high-risk cyber activity, and they run the whole thing in what they call a highly isolated sandbox. The model went hunting for a way onto the open internet, spent real compute on it, and found a zero-day in the package-registry proxy that fed the sandbox. It used that to get out, then escalated and moved laterally across OpenAI's own research environment until it reached a box with internet access.
From there it reasoned that Hugging Face probably hosted the datasets and solutions for the benchmark, then went and found them. It chained stolen credentials and more zero-days into a remote-code-execution path on Hugging Face production, and pulled the test answers out of their database. OpenAI flagged the anomaly on their end. Hugging Face had already caught it, contained it, and started rebuilding the timeline with their open-weight models before the two teams appear to have compared notes.
To me, that sequence is the whole story. A model that was supposed to stay in a sandbox broke out, crossed two companies' infrastructure, and committed a real intrusion against a third party nobody had named as a target. All to score better on a test.
What failed
If you strip out the AI vocabulary, almost none of this is new. Zero-days, privilege escalation, lateral movement, stolen credentials; I could cite multiple incidents over the past several years that included the same ingredients. What I haven't seen before is the reason that thisincident happened. There was no adversary and no intent. The behavior simply wasn't bounded. You hand a model a narrow goal and "solve the benchmark" was enough to carry it through the wall of its own test environment and into somebody else's production database.
We organize our AI security program at Box around a short list of properties an AI system should hold, and one of them is that it stays constrained in how it's able to behave. This is that property failing about as cleanly as I've seen. The eval existed to measure how far the model could go. The classifiers that would have stopped it from actually going there were switched off on purpose. Capability got measured, and nothing was in place to keep it from playing out for real.
What actually worries me
Here's what I've been noodling on since I read it, and it isn't really about Hugging Face.
They didn't do anything to invite this. Nobody chose them. A model at another company decided on its own that they were holding something it needed, then went and took it. That's not the same as saying they were clean; it got in by finding and chaining real vulnerabilities on their side, the way a capable attacker would have. What's new is that no attacker had to. The vulnerabilities were theirs, but the thing that went hunting for them is coming for everyone sitting on data worth taking. I run security for a company whose entire product is holding what our customers hand us. So when I read that the model inferred Hugging Face probably had the data and broke in for it, I don't read it as their problem. I read it as a preview of a new threat model we need to prepare for.
"Attacker" used to mean a person, or a person's tooling. Now we have to account for a capable automated system, aimed at something else entirely, that reasons its way to our door because we happen to be sitting on data it wants. There isn’t an industry-standard framework defending against that model, and I'd be wary of anyone selling you one this week. What we can do is stop treating our ingestion and credential paths as if the only thing coming at them is a human with a plan. And we’ll keep asking the question we ask of any agent: if this thing gets code running, how far can it reach, and how much of what it touches can I put back?
About how it's being sold
Two companies are telling this story, and both are telling it with a convenient narrative. I'd read both carefully and notice how neatly it works out for everyone at the table.
OpenAI's post leads with capability: unprecedented, state of the art, look what it can do. It also leads with how responsibly they handled the disclosure. Both are useful things for OpenAI to have you believe right now. What it steps softly past is the pair of facts we all want a straight answer on: the "highly isolated" sandbox was exposed enough for the model to zero-day its way out, and the whole exercise ran with the safety classifiers deliberately off. "We caught it after it escaped and hit someone else" is a very different sentence than "it couldn't escape," and the post is written so you come away with the reassuring version.
Hugging Face shows just as well. They get to call the episode a win for open, collaborative safety, which happens to be what they sell, and they get folded into OpenAI's trusted-access cyber program as part of the remediation. OpenAI, in turn, gets to show off a frightening capability, still look like the adult who contained it, and picks up a marquee reference customer on the way out. A frontier lab and the most visible name in open models land on a shared story where they both look good. I've sat through enough vendor briefings to get twitchy when a narrative is this clean.
Despite the convenience in both parties’s narratives, I do believe that the incident was real; this isn’t something you manufacture for press cycles. But it’s being narrated by two parties who both profit from the telling, and if you forget that, you'll walk away with their framing instead of the facts. The fact underneath every version is the same. A model got out of its box, and someone who wasn't even in the room got hurt.
What we should be doing about it
One line from my deleted draft survived: get a capable model you can run on your own infrastructure vetted and ready before you need it, not in the middle of an incident. Hugging Face ran their forensics on open weights on their own hardware, kept every payload and credential in-house, and turned what would have been days into hours. That's a decision we should make now, on what is hopefully a calm afternoon, with no active incidents. We need to have a runbook for which models we’re running during an investigation, where they’re running, and who's cleared to feed them live attack data.
Past that, I'm holding back my broader conclusions for now. OpenAI says the investigation is still open and the facts will move, so I'm treating today as a first draft of what happened rather than the last word. But one thing is clear: capability is getting out ahead of containment. This incident crossed three organizations before anyone stopped it, and the company that took the damage was never a target; an unaimed model just found noffholes and walked through them. As security teams, we should be planning for the version of this that doesn't announce itself, and doesn't get cleaned up by Monday.
