|
Just a few more thoughts on that OpenAI rogue AI agent an wrote about earlier this week.
As I noted in my originl story, the tech bros treat AI as a game and have little to no care about the enormous risks they are creating.
And you need to look carefully at what happened. An AI cannot "decide" to hack an external target without it being deliberately instructed by a human. And what happens if they also deliberately made the sandbox environment hackable, or if their internal process was deliberately made to be sloppy?
There are many human factors that are needed to lead to an incident like this, rather than labelling it as "autonomous".
And the news yesterday bears me out.
OpenAI had removed the safety filters for an in-progress model, locked it up in a sandbox, and told it to solve the ExploitGym problems.
Note to readers: without getting into a long eplantion, "the ExploitGym problems" are a benchmark for evaluating models on their ability to turn vulnerabilities into concrete exploits. "Release the hounds!"
Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead.
Note to readers: Hugging Face is important because it acts as the central open-source hub and collaboration platform for the entire global artificial intelligence community. I use it all the time. Often called the "GitHub for Machine Learning," it democratizes AI by providing free access to hundreds of thousands of pre-trained models, datasets, and interactive tools. It's a goldmine.
OpenAI’s sandbox used the now common pattern of allowing access to an allow-list of sites related to installing additional packages via a controlled proxy. In OpenAI’s own words:
While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.
So step one was to let it break out onto the public internet. The model then broke into Hugging Face to find the answers:
After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. It found the solutions to launch a cyber attack.
So the results show that autonomous exploit development by frontier AI agents is no longer a hypothetical capability. While current agents are not yet reliable across all targets, they already can exploit a non-trivial fraction of real-world vulnerabilities, including very complex targets. This rapid emergence is itself a central finding, showing that capabilities that would have seemed implausible are now present in deployed frontier models.
An important detail here: this isn’t about discovering vulnerabilities; it’s about being able to take those vulnerabilities and turn them into working exploits.
And the asymmetry is increasingly frustrating. Hugging Face, faced with an accidental and aggressive attack from one of OpenAI’s models, were unable to then turn to OpenAI’s models to help them fend off the attack.
The frontier models we have access to are increasingly being constrained in how much they can help us protect our software, heavily influenced by the U.S. government’s ongoing threat of export controls.
Meanwhile open weight models from China such as GLM-5.2, Kimi 3 and the new Qwen 3.8 Max have none of these restrictions - and any restrictions that do exist can likely be fine-tuned out of them by modifying the weights.
| 
