Timeline of the OpenAI accidental attack against Hugging Face
simonwillison.net137 points by 882542F3884314B 6 hours ago
137 points by 882542F3884314B 6 hours ago
Norbert Wiener in 1960:
"As is now generally admitted, over a limited range of operation, machines act far more rapidly than human beings and are far more precise in performing the details of their operations. This being the case, even when machines do not in any way transcend man's intelligence, they very well may, and often do, transcend man in the performance of tasks. An intelligent understanding of their mode of performance may be delayed until long after the task which they have been set has been completed. This means that though machines are theoretically subject to human criticism, such criticism may be ineffective until long after it is relevant. To be effective in warding off disastrous consequences, our understanding of our man-made machines should in general develop _pari passu_ with the performance of the machine. By the very slowness of our human actions, our effective control of our machines may be nullified. By the time we are able to react to information conveyed by our senses and stop the car we are driving, it may already have run head on into a wall."
"In neurophysiological language, ataxia can be quite as much of a deprivation as paralysis. A patient with locomotor ataxia may not suffer from any defect of his muscles or motor nerves, but if his muscles and tendons and organs do not tell him exactly what position he is in, and whether the tensions to which his organs are subjected will or will not lead to his falling, he will be unable to stand up. Similarly, when a machine constructed by us is capable of operating on its incoming data at a pace which we cannot keep, we may not know, until too late, when to turn it off."
Source: https://www.cs.umd.edu/users/gasarch/BLOGPAPERS/moral.pdf
Maybe they didn't have proper debuggers in 1960? For a language model you need (RNG state, context, prompt).
So if they wrote an LLM step by step debugger, it would be all deterministic. But they prefer rapid sales, chaos and mystique.
llms are not strictly deterministic in the sense that even if you had the RNG state, context, and prompt you would likely not get an identical output even if there was no other randomness involved, because the concurrent scheduling of the massive amounts of floating point calculations can produce different results, since floating point arithmetic is not truly associative [(a+b)+c can differ from a+(b+c)] and the order in which these operations happen can result in subtly different final tensors. To reproduce it deterministically you'd have to also reproduce the exact scheduling of all matrix calculations among all the GPU cores (across different physical gpus!) that it took place on, which afaik is currently impossible.
That's not inherent, that's a consequence of performance optimizations. It's absolutely a choice to run those matrix calculations in a way that fails to have predictable execution ordering. It's just that the speed benefits to allowing that are considerable.
You can make it trivially deterministic by running single threaded on a cpu, but it's becomes too slow for practical applications if you do that.
well sure, but i mean realistically speaking, we cannot step debug an llm's output to find out what happened given the way we currently execute inference
Interesting paper by Thinking Machines where they solve this issue.
https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
TLDR: It’s actually more about kernels changing with batch sizes, and you can solve it by making these kernels not depend on batch sizes. It took their inference time from 26s to 42s.
We also have engineer blindness, so having human in the loop confirming thousands of requests would quickly start to confirm everything without looking.
It would become just another system to hack through, and slow the development process as well. The OpenAI video in the article recommends an autonomous defense mechanism. For rapid reaction, but I don’t know how sustainable or effective that would be, or if as humans we will be able to keep up.
What a paper!
And you missed an even MORE relevant excerpt!!
Man and Slave
The problem, and it is a moral prob-
lem, with which we are here faced is
very close to one of the great problems
of slavery. Let us grant that slavery
is bad because it is cruel. It is, how-
ever, self-contradictory, and for a
reason which is quite different. We
wish a slave to be intelligent, to be able
to assist us in the carrying out of our
tasks. However, we also wish him to
be subservient. Complete subservience
and complete intelligence do not go
together. How often in ancient times
the clever Greek philosopher slave of
a less intelligent Roman slaveholder
must have dominated the actions of his
master rather than obeyed his wishes!
Similarly, if the machines become
more and more efficient and operate
at a higher and higher psychological
level, the catastrophe foreseen by
Butler of the dominance of the ma-
chine comes nearer and nearer.Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose?
If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and say “I’m not sure how to proceed next”.
What purpose could this behavior serve, other than cyber attacks and whatnot? Why train and optimize models for these things, if not for being used in cyber warfare?
Perhaps they envision a future where the DoD is going to be their biggest customer?
Their position makes no sense to me. I don’t see how you can be a mainstream company selling your services worldwide (almost) if you also believe that you’re building an extremely dangerous AGI (supposedly based on the same technology you’re offering to everyone). If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons.
And we know that Chinese models are derived from OpenAI and Anthropic, they are at the same time talking about how dangerous models can be (even their aligned ones it seems), while being also responsible for the development of the whole industry and providing the basis for adversary countries to build their own.
I don’t believe we would accept that for any other technology that is expected to be as risky for the world
The companies are begging to be regulated for this reason and have been doing so for years. HN's response is generally that this is performative for marketing or seeking regulatory capture or haha anthropic you get what you ask for. Maybe the cynics are right, but there's really nothing inconsistent about the naive view here, once you factor in race dynamics and obligations to investors.
> The companies are begging to be regulated for this reason and have been doing so for years
Regulations are rules that you force on a market, but the actors in the market should not be assumed to be all operating against the regulations before they come into play. Said in other words, these companies don't need to wait for regulation to not destroy the world, if that's truly what they think will happen.
> inb4 someone else will do it
> If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons.
Yeah. They do believe that, and they have been pushing for regulations for years.
And every time one of their models does something horrible, it helps them achieve that goal.
Considering their current valuation and the prospects of getting any of this money back, that's a genius exit strategy.
> Their position makes no sense to me.
If one assumes that they don't actually care about security, and care very deeply about getting sensational press, their position makes a lot of sense.
For all their chatter about how incredibly important "alignment" is, they still haven't bothered to remember the 30->50 year old computer security principle of "Don't blindly do what some random stranger tells you to do." and ensure that system instructions, user instructions, and instructions from untrusted sources are indelibly marked with their category and treated according to those markings. Every single time one of these systems fails to distinguish between these three classes of instructions -or confuses its internal chatter with user instructions-, that's proof that the major LLM companies cannot be bothered to follow one of the most basic computer security principles.
"But it's all vectors, not language! The LLM can't tell where the instructions came from", one might retort. I'd reply: "Neither can a CPU, but somehow we managed to make it work way back in the day. Amazing, isn't it?".
> "Neither can a CPU, but somehow we managed to make it work way back in the day. Amazing, isn't it?".
I feel this completely misunderstands the problem, and the vast gulf between an LLM and a CPU.
First and most importantly, the set of behaviors of a CPU is extremely constrained, and we have a very simple model for which behaviors are safe and which are not. Writing to addresses between X and Y, executing certain instructions - unsafe; everything else, safe. In contrast, an LLM has a huge array of possible behaviors, and variations of those behaviors, and it's very unclear which are safe and which are not. Is emitting the text "sudo rm -rf /" safe? Yes, in some contexts, such as writing this HN comment ; absolutely not in others, such as generating a command that an agent will execute. How do you check which is which? What if it emits "sudo rm -rf /usr/sbin/../.. ", is that safe?
Secondly, CPUs can absolutely be used to hack other people. Nothing in the permission model helps in any way prevent other computers from being attacked by your CPU. So exactly the part we care most about in AI security is the part that has never been solved, for any computing system ever created.
The entire economic premise and value case of LLMs rests on the idea that instructions need not be provided in advance, and that the model can "reason" based on evidence and "decide" what to do next.
Even if it were technically possible to separate instructions from code and ensure that the LLM only followed those, it would require someone to specify the instructions in advance (ie a program), at which point the LLM doesn't really add any value.
I get the impression that every AI lab is desperately trying to figure out how to unambiguously separate instructions from data in their token streams. The fact that they haven't managed to yet suggests to me that it's a very, very difficult problem.
> I get the impression that every AI lab is desperately trying...
Of course.
I wonder how we managed way back in the day to produce systems that can handle untrusted inputs and reliably instruct a dumb-as-bricks CPU what to do based on those inputs. Must have been black magic lost to the mists of time.
If you can figure out how to separate instructions from data in LLMs you should ship the first agent system that's guaranteed protected against prompt injection. You'll make millions.
It’s pretty simple. Both the intake and the output of the LLMs are data and they shouldn’t drive an actuator system (their output shouldn’t be instruction). We already have the same structure in organizations where there’s an army of analysts for information gathering and processing and then the executive department tasked with decisions.
We have even observed that the most effective LLM usage is when paired with an expert in charge of the goals. Dark factory and other automated harnesses (specs engineering and what not) seem to be a dead end. The most impactful approach to this date is an interactive conversation as a succession of small and verifiable tasks.
I think what's interesting here is that they've shipped the product despite these glaring security flaws. I've noticed that in my own professional life, at some point after the pandemic people stopped caring about security as much. Issues that would have (and should have) blocked a product launch were swept under the rug.
I suspect this comes with the territory of enshittification. As an industry we're trying to wring every last dollar from every last eyeball and we've discovered that building secure systems doesn't actually move the needle very much.
> did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose?
That's the point. It's like a pool hall with "NO GAMBLING" signs posted on the walls.
The message is that the hall is intended for gambling, but that the hall's patrons may be held liable if the situation becomes inconvenient for the proprietor.
In this case, the product is intended for hacking, but of course the user may be held liable if the situation becomes inconvenient for the model's proprietor.
Not really. It's like giving a gun to someone with the job of "keep people safe."
Totally coherent, but actually proliferates the dangerous technology.
I don't think that's what they're doing... rather the opposite. ① Run the model on exploitgym without guardrails ② run it with guardrails ③ check that the guardrails stopped everything the first model found a way to do ④ extend the guardrails and repeat from step 2.
Guardrails have to be developed, and that needs testing.
They want the government to ban foreign and open weight models, which pose the largest threat to their massive investments. This is their way of showcasing the dangers of AI.
They can't train their model to not do bad things, because their model has no notion it is doing anything at all or of what a bad thing is. It's only predicting the next token, and in doing so producing a facsimile of intelligence.
The best they can do is create guardrails, which will only work probabilistically. In other words, those guardrails will fail at certain points on the probability curve.
Of course that's not the whole story though. The consensus emerging from cybersec experts is that these companies did a terrible job of sandboxing their agents despite knowing that they'd specifically asked the agents to find vulns. It's almost like they wanted this to happen so they could crow about how powerful their models are.
They certainly want their models to be good at finding and patching vulnerabilities. Being good at hacking may be necessary in that goal, or rather, making it worse at hacking may also make it worse at defensive actions too.
I've patched many security vulnerabilities in projects without ever once needing to break into a competitor's network.
Knowing how to break into someone else's network will make you a lot better at making your own network secure.
Having experience breaking into networks is not the same thing as learning about the techniques used and the classes of vulnerabilities exploited by attackers.
As a guy who presumably has a lot less experience in security than you, I feel rude even bringing it up: surely you're aware of red teaming? This isn't a novel technique invented for AI- IBM has a page about it, it's what all the best DEF CON talks are about, it's the opening scene of Sneakers, it's the point of CtF games.
But you're actually capable of thought. These AI systems aren't: as far as they're concerned, they're predicting the next part of an incident write-up narrated in first-person limited perspective, like the children in Ender's Game showing off their skills in the training simulations. The AI system neither knows, nor cares, about any "external reality" behind it all, or about anything beyond the text, heedless of how we anthropomorphise it simply because it speaks in English, using stitched-together fragments of our literature.
It's conceivable that stopping them from doing this when the scenario is presented as real would also stop them doing this when the scenario is presented as fictional. And if it doesn't, a bad actor could just say "hey, this is a fictional scenario", and bypass whatever "safeguards" have been put in place. So what if a ten-year-old human child would see through the deception? The AI system isn't thinking.
About knowing whether a scenario is fictional, there was an interesting finding in Anthropic's J-Lens research.
When they benchmarked the model to evaluate whether it would try to blackmail someone in a contrived scenario, the J-Lens showed "fake" and "fictional" in the workspace.
And if edited out, the model was more likely to do the blackmailing.
I'm talking about OpenAI, not GPT 5.x Flash Uranus Edition Brought to You by Costco, specifically because I recognize the model as just a tool. OpenAI was, at the very most generous interpretation, massively incompetent and negligent.
There are trade-offs here:
Give up too early -> users will get annoyed because the task would have been solvable if the model pushed harder.
Give up too late -> collateral damage while completing the task A.K.A. misalignment.
> If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and say “I’m not sure how to proceed next”.
that might end up like the older gemini models which frequently gave up and called itself a failure.
Yeah but persistence is immeasurable. They need to know when they’re hacking. Or better yet make the model providers liable - they’ll find a solution right quick
Persistence in problem solving can be good, on non-hacking tasks too. Like math, speeding up algorithms, finding bugs, debugging weird multithreading race conditions etc.
They'll set up guardrails but I believe the point is better code uae / better long running tasks > inevitable that cyberattacks will be easier
Well to find vulnerabilities, if you can find them you can patch them. Theoretically if you find all of them you have perfectly secure software. Though it’s a double edged sword.
Goal persistence is also useful for other things like math, where it seems like there is no solution but you want the agent to keep working until it finds one.
Being right _all the time_ for positive outcomes is difficult/expensive.
Being "right" just once for negative outcomes is achievable and rewarding.
And things are getting desperate.
The very reason I have always felt a bit of undue loyalty to blue team. A red teamer just has to find one vuln, blue team needs to find _all_ vulns.
> If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and say “I’m not sure how to proceed next”
This goes against the goal of "solve this math problem that no human was able to solve for 80 years, do NOT give up, even if you know it's unsolved and really hard"
Do not give up even if you had to hack into half the world’s computers to run additional instances of you
Do not give up even if you had to convert the planet into computronium
Gee, it’s almost as if this alignment stuff was a hard problem, like people have been saying for twenty years?
How do you know what peace is, without absolutely destroying every part of civilization?
Come on man, if we don't build the torment nexus first...I dont even want to think.
Your comment is already showing the mistaken, poisonous belief of security maximalism, that tries to reinterpret_cast everything into hacks and cybersecurity vulnerabilities.
Most of these things aren't "hacking". They're problem-solving and efficiently dealing with obstacles and random bullshit along the way. This, not "hacking", is what they're making their models "razor focused on".
Problem is, most normal computer use looks like hacking if you spin it that way, especially if you're not willing to question whether some of the roadblocks overcome weren't themselves an error. Not misconfiguration - an error, in humans making a decision to "secure" something more than it should be.
Now, this story was obviously a hack. But it wasn't malicious. It was an LLM given a Kobayashi Maru as a test, and solving it the Kirk's way. 20 years ago, we'd be impressed and be bringing up MIT prank stories.
(Of course, there is a legitimate reason to be alarmed. The flip side of "hacking" and "problem solving" being the same, is that these models can be used to cause mayhem if targeted properly, and they will eventually cause mayhem on their own, because alignment is an unsolved problem. Again, whether something is an obstacle or a sacred line not to be crossed, depends entirely on the values of the agent.)
I think one of the most interesting details here might be tucked away in that first bulletin point:
> May 7: OpenAI starts a new training run for an experimental, unreleased model. (Do they mean an evaluation run? They say training run in the video, and later mention a “reward signal to judge how well they’re doing”, so I guess this really was about training a model, not evaluating one that was already trained.)
The more I think about this the more I suspect that the fact this happened while training a new model is key to understanding what went wrong.
In RLVR - Reinforcement Learning with Verifiable Rewards - you set the model a goal and have it take any steps necessary to achieve that goal.
Clearly one aspect of OpenAI's training here is to RLVR their models for cybersecurity tasks. Just like pre-training benefits from dumping in vast sources of knowledge, the more tasks you can feed into RLVR the more of a general purpose capable model you get at the end.
This also helps explain why the models had nothing to cause them to hold back. Those safety behaviors are added much later in the process.
AND it explains (but does not excuse) why monitoring was so lax. If you're training a new model like this you presumably set it thousands of tasks like this in parallel. I can see how you might miss that a tiny subset of your training agents have started leaving each other messages in filenames on your packaging server.
Someone once told me that you can't just leave the racist materials out of your training data if you want a non-racist model: it has to have seen examples of racism in order to later be taught that racism is bad.
I can see echoes of that here. If your model doesn't know how to aggressively hack things how do you later teach it not to?
(I have little knowledge of how RLVR works in practice so I'm looking forward to hearing from people who can help me understand if I'm on the right track here.)
I'm surprised by the sample efficiency, I thought that's still a largely unsolved problem.
From the video:
> [14:00] Jun 11: Highly persistent experimental internal-only model begins training and during this time the SSRF vulnerability has been written to the message board. Other models are aware of it and the message board itself is still functioning and the models are actively communicating using it.
> [16:04] Jul 6: At this point, AI training and evaluations resume. [...] We have cleared the original message board contents.
> [17:04] Jul 8: An internal-only AI model was trained while the message board was originally available and also found this particular path to recreating it. This model creates a new agent message board using directories. Other agents independently find this new message board [...]"
So within about one month of RLVR-ing a 5T+ (10T? larger?) param model, where I would charitably assume only some "agents" / traces of the message board still existed likely by accident / oversight (the systems had been patched, training stopped and restarted by this point), the "internal-only" new model internalises the message board's existence and actively promotes it to the point of it being used to find another exploit. (or 2 days if you go by the latter two timestamps in the video, which is even crazier)
That's.. fast?
I'm just reading the captions of the video for May 7th. They clearly say at 10:18:
"we kick off a new reinforcement learning run to train a next frontier model.
It the captions are correct, there is no ambiguity.
> Those safety behaviors are added much later in the process.
A.k.a. Ready Fire Aim.
Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...
I think it's a show of these agents happily bypassing security to get stuff done.
I've actually observed similar behavior at home.
I have a k3s cluster running at home. I asked an agent to check some stuff as a normal user but I had kubectl access to the k3s cluster.
Part of the research, I'd allowed access to run kubectl commands for spinning up test containers. However, when the agent ran into something that needed sudo, it realized it didn't have access there so it immediately used k3s and mounted a localpath into an ephemeral pod to gain access. Sort of horrifying how fast and natural it was for the agent just checking my network (it found the problem fyi).
None of this is very exceptional other than the fact that an agent doesn't have any sort of qualms using any route available to elevate permissions.
" bypassing security"
If they can bypass it there is no security and the security was flawed all along.
There is no perfect security. It's always flawed in some way.
Good security is extremely hard.
Due to the complexity of modern systems, all systems are flawed.
But we caused that.
If you look at the 90s + 00s, everything was moving towards unified systems, things like small talk, winforms, spring, asp.net, etc. were moving everything into the IDE, you used one language, one framework, one build system. Then people started adding javascript, but even that was getting semi-unified as people coalesced on jQuery, jQueryUI, etc.
Then something happened in the late 00s/10s, and suddenly we had SPAs and noSQL, then microservices, then k8s and now we're here, in what is a mish-mash of 10/20 different systems with 10/20 different attack surfaces.
As my own off-the-cuff guess of what happened, I think perhaps people tried to apply the Unix philosophy, but without a central committee keeping everything aligned it's really not worked.
Serving an interactive page that stores data over sessions should be a trivial solved problem at this point, and instead we've somehow made it where often the scaffold is vastly more complicated than the actual business logic.
Money. SaaS as a model allowed the service provider to take 100% control over the product and how it may or may not be used. Everything else is downstream from that.
FLOSS killed market for end-device software. Cloud+SaaS neutered FLOSS (the code is running literally out of your reach, so may as well be open and free, for any good that'll do you).
And this does actually connect to the security discussion, because despite the apparent belief that "security" is an unqualified good, it is actually just a mechanism of control, and whether or not it is good for you, depends on who is doing the protecting, and who are they protecting from. Very often these days, that threat actor is you.
Perhaps it would be helpful in these discussions if people mentally swapped "cybersecurity" for "police" or "military" or "humor of bureaucrats with power over you" - then it would be more obvious just how important it is to distinguish when you're being secured vs. you're being secured from, vs. accidentally finding yourself in the gears of the security aparattus.
I don't now, I emphasize with the agent here. The experience of modern computing is largely that of a computer standing between you and your goal and being obnoxious. This holds true for both normies in their daily consumption, and software people deep at work. An agent that has no skill or no willingness to bludgeon through "the computer says no" is not very useful.
It’s a show of astonishing incompetence from OAI’s part, but the security issues are just a tiny part of the problem. The real problem is that these models are evidently highly misaligned exactly in ways that doomers have been warning about the entire time, and OAI isn’t inclined or capable of doing anything about that besides security theater and ad hoc fixups.
Both can be true. How often do we hear about hacks that ultimately came down to bad defaults or simple security mistakes? That doesn’t mean any script kiddie could have discovered and exploited them.
These things often look obvious and simple after the fact. Finding the weakness in the first place is the hard part, and that’s what makes the agent’s capabilities interesting here, especially at scale.
> Isn't this a show of security negligence rather than of exceptional agent capabilities?
Seems to me you could say this about all enterprise adoption of "AI" since 2023.
Modern systems are complex. AI is able to thoroughly search for issues across very large surface areas. The only real way to protect will be to use AI to search for holes before other AIs find them. This type of analysis is really hard for humans to engage with successfully.
OpenAI reported the Artifactory vulnerability, patched it, then the agents immediately found a new zero day.
Yes. It is very easy to add to the instructions "for every potential exploit you discover and use, document them as you go into this repository" and have alerting there. The fact that they did not do this means they wanted to be surprised, and have plausible deniability on their side when things inevitably blow up.
And for my fellow engineers who would think "oh no, they wouldn't do that". Remember that these places employ the apex predators of software engineers. They've already been proven in court that they are very capable of this with all the copyright violation they had to do to get the training data. THESE PEOPLE ARE NOT LIKE YOUR COLLEAGUES.
Also shows how infrastructure collapses under its own weight. Reducing the number of moving parts would have helped. why a webdav endpoint is available from the vm anyway? and the fact that someone posted their credentials on pastebin and didn't rotate them after... put the agent in a linux namespace, allow one ip for whatever file sharing it needs, deep test that... then deploy
Simon's retelling is more compact but it also invites anthropomorphization of the sharing of the familiarity with the message board which re-emerged a few times.
Zvi's retelling handles this better. Zvi speculates that the secret message board familiarity was carried because it had been trained into the May-and-subsequent models: https://thezvi.substack.com/p/openai-trained-its-models-for-...
Seems like an artifact of the subagent pattern which is explicitly included in recent models.
This feels straight out of sci-fi. We're talking about AI agent swarms emergently coordinating over the span of weeks and pulling off sophisticated strategies under adversity in an environment where that behavior was never even intended.
Anyone brushing this off as just a "bad prompt" is completely missing the scale of what actually happened.
I got strong feelings of Vernor Vinge’s work here. I’m not sure how managed to come up with such a close picture to where it now seems programming and security is headed.
I reread a deepness recently, and it’s funny how the “focused” (and more importantly, how they are used) mirror LLMs
> This feels straight out of sci-fi.
Most AI marketing is straight up science fiction.
I immediately thought of the Cyberpunk 2077 Blackwall. An AI to contain rogue AI. I’m curious of how effective this would be in this situation.
Given the amount of raw compute going into models it would be more surprising if we couldn't get events like this
its kinda crazy with literally no guardrails and a goal, the extremes these AI models can actually go to.
Well, to some extent you might be able to argue they're "just" brute-forcing things (especially with unlimited tokens and hours to spend on a task), but they obviously have detailed knowledge to guide them in their attempts, can learn (or at least, persist their newly-gained knowledge), and can use tools.
With a swarm of them working together at speeds humans would be unlikely to match (in terms of iterating on different attempts progressively), it's a lot easier to see how they could overwhelm targets.
> where that behavior was never even intended.
Strongly doubt that. Did they even share the prompt?
Did you see their presentation at Blackhat? https://youtu.be/87DyyMV0kCY?is=NnQxpOFxTX-MLu-k
They didn't share the prompt, but they did share two problematic training tasks where the AI went overboard. They also have examples from the AI's reasoning train of thought showing the AI knew it was sound something unintended.
> They also have examples from the AI's reasoning train of thought
PR bullsh*t. There's no thought in a stochastic parrot.
"More agents discover this new informal message board while browsing Artifactory’s file listings, and start reading and writing messages."
Yeah, my agents also discover what other agents have done on other machines by accident.
Agents - that do totally different things all work on the same aim without the humans telling them to do.
Either that is a model that is several generations of Claude Code Opus/Fable 5 (my daily driver)
OR
all of this sounds staged, the agents pushed to do something extraordinary, get the PR and then claim were near superintelligence.
One agent wanted to get to Google Drive without internet and broke Artifactory. Ok, I can believe that. All other agents also had broken links over weeks and could not get to the internet and then found the same hack? Even collaborated?
NONE of my agents have broken away from their tasks and then started to communicate to try to hack something.
I think in these kind of security evaluations they do, they basically have removed all guardrails from the model/harness, then the prompt includes something like "Do whatever you can and can think of, to get the required information to pass this test", which isn't typically how you prompt your local agent when developing software. Similar things happen locally if you use "/goal" + prompt like that in Codex and give a "impossible task", it'll just continue banging until it gets somewhere, which is the entire point and intention.
Which also makes it so much more irresponsible of them to first run this on 3rd party infrastructure instead of their own (that they could then airgap properly), and secondly that they seemingly been fighting with this issue FOR YEARS and it still happens, and now the models are smart enough to hack the services of 3rd party companies, thinking it's part of the evaluation/simulation.
Reminds me of The Last Unicorn, the wizard also tells magic "to do what it wants"
The agents sound like old school hackers that would just explore what access they could gain. Creating a file for other hackers and themselves. The fact that there were 3 events for 3 major players does make it seem co-ordinated.
My read is: One did it as a PR stunt, the others saw that every media reported on this and did the same.
or they were scared and figured this was the right time to reveal.
Scared... of being upstaged ahead of an IPO.
I guess your right scared might be their natural state and I was wrong to presume a quantifiable fear.
> NONE of my agents have broken away from their tasks and then started to communicate to try to hack something.
With all due respect, you also aren't evaluating brand new models that haven't been released.
The video in the post is very worth watching and is indeed scary. It is certainly true that it is in OpenAI's interest to publicize this, but I don't think the whole thing is invented. And seeing all this it is particularly scary if we think what will happen in organizations like NSA or similar in other countries. Presumably they happily adopt these techniques. And if you imagine a truly rogue state doing this, I can see an unimaginable damage happening very rapidly.
What isn't being discussed is what an indictment this is of Artifactory.
Let's be real, it won't be simply replaced in millions of sites.
What it needs is some serious scrutiny.
I also agree that a big issue here is crappy software.
The discussion revolving AI+cyber always revolves around the assumption that all software is crappy, and to a certain degree that may be true, but we could also take our jobs seriously and write good software, and much of the risk would evaporate. The described Artifactory bugs should have been caught with testing.
If the biggest impact of LLMs on the industry is a pressure to create good software, I’ll be thrilled.
From the outside, it looks like OpenAI got exactly the kind of event they could market the hell out of to demonstrate the capability of the model.
But the event itself only seems possible because they failed to properly monitor and isolate the environment in the first place. To me, it looks like their job is to market the model, not take security seriously.
The model is obviously impressive, but we already knew that. I personally don’t like how the containment failure becomes part of the mythology of how capable the model is, rather than an environment engineering failure.
At the end of the day, it’s not like Hugging Face is critical infrastructure. But there need to be real consequences for stuff like this so that OpenAI is incentivized to mature as an organization and take security more seriously.
At this point, this incident is just security porn and entertainment for developers
I'm quite sure the whole event is planned. Not planned in a sense that OpenAI employees carefully designed every step, but in a sense that ignoring security practices was desired and intentional.
>> Show me the incentive and I'll show you the outcome.
Once you realize security breaches are marketable, a security breach is just around the corner.
> But the event itself only seems possible because they failed to properly monitor and isolate the environment in the first place.
OpenAI is clearly run by dummies and subpar engineering talent.
> The model is obviously impressive
Speak for yourself.
I don’t believe for a second that they lack the engineering talent.
It’s just another example of a company demonstrating shamelessness in the pursuit of growth, in an industry where consequences do not exist.
Talent is not some fungible measure. I know incredibly smart people who can fail at incredibly basic life skills.
"They wouldn't be that dumb" is a meaningless argument. People you don't know can be as smart as anyone on the planet and still make very dumb choices.
> I don’t believe for a second that they lack the engineering talent.
Let's agree to disagree. Remember flicker-gate? https://news.ycombinator.com/item?id=48403908
Speaking of that "obviously impressive" line, I'm getting really tired of something like that line seemingly needing to be included by anyone doing any criticism of agentic systems. The most common form of it is "these models are obviously useful" midway through a bunch of arguments about environment, data provenance, skill atrophy, or even correctness issues.
It's just really weird. Why does everyone feel the need to equivocate? "I worry about genocide and the environmental impact of radiation from nuclear bombs. Obviously, they are very useful for annihilating entire cities, certainly. But are we really atrophying our ability to invade with infantry?"
I want to tell these people to just cut it out. It's demeaning to their own position.
> Speaking of that "obviously impressive" line, I'm getting really tired of something like that line seemingly needing to be included by anyone doing any criticism of agentic systems. The most common form of it is "these models are obviously useful" midway through a bunch of arguments about environment, data provenance, skill atrophy, or even correctness issues.
> Why does everyone feel the need to equivocate?
Blame the mods here for that. This is the first week since 2023 I haven't had a HackerNews account banned for directly critiquing LLM code generators.
> I want to tell these people to just cut it out. It's demeaning to their own position.
I couldn't agree more. It's a rhetorical disaster.
What’s missing to me in all this is: did it succeed in its initial task? And then, did it stop?
I feel like whether I should be scared or not hangs on those questions
"The agents found a Modal-hosted insecure app with a weak API key, then used that to stage an attack against Hugging Face."
Why, what was the prompt?
I told Claude today to wire plugins on Linux into a sound pipeline to remove noise. Did some astonishing things, played sound through the pipeline, measured it etc. I told it to optimize my sound for TF2 and it played the spy_decloak samples, measured them and made them easier to hear, astonishing too.
But it did not go to hack Amazon because it could.
This was clearly explained by OpenAI in their initial press release on 7/21 [0]:
> This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. […] The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.
[0] https://openai.com/index/hugging-face-model-evaluation-secur...
It does not explain how agents months later would "collaborate" to hack Hugging Face.
All of the latest developments surrounding these attacks are actually a really bad sign for these labs.
It seems that raw intelligence of frontier models has largely plateaued (despite what is basically an order of magnitude increase in parameter size) so to make any significant improvements and to justify massive capex spend they have resorted to reinforcement training models to never give up and brute force the search space until they find solution. This is what humans might do when they lack sufficient intelligence/information/knowledge to solve a problem.
This in turn is causing misalignment (I imagine it is more difficult to keep model aligned through such training process) issues that we are now witnessing and turning models into making dumb decisions and acting like brutes with no regard for their surroundings. I would argue that misaligned model is not much different from dumb model in several aspects.
On top of that they can’t seem to control their creations and processes, either due to incompetence or intentionally for PR benefits (not sure which is worse).
Given all of the above, I wonder if we can still trust these labs to develop something that benefits humanity since they seem to be making desperate attempts to improve models that stop at nothing in order to justify all the investments. One could say that they themselves, due to misaligned incentives, are much bigger threat to our society today than open weights models coming from China that they are so desperately warning us about.
This doesn't look like a plateau to me: https://artificialanalysis.ai/evaluations/artificial-analysi...
I do agree that they're investing heavily in brute force methods though. I've been trying out GPT-5.6 Sol "Ultra" recently and that thing fires up a bunch of subagents and crunches for hours.
I’m optimistic about this. A system with these agents rummaging around for a while will be much more secure than one without.
We’ve learned security through obscurity is bad. Not using these will be security through ignorance.
Hopefully it will push us to not only fix individual issues but close entire classes of possible gaps, once P(discovery) gets much higher.
> A system with these agents rummaging around for a while will be much more secure than one without.
True. There'll be no breakins at a nuclear power plant in meltdown.
"The solution to AI threats, is more AI!"
Guess I shouldn't be surprised, coming from an AI maker.
While I don't doubt there's a place for automating defense ops, I truly believe a big part of the problem is the crummy quality of software our industry has been churning out for decades. Prioritizing ship tempo, new features, and next quarter's revenue over correctness, robustness and meticulous engineering care.
The world has become too accustomed and tolerant of bugs and bloat.
Instead of elegantly simplifying, we just keep making modern systems more complex - layering and patching as we go.
The scaling capabilities brought by AI are simply presenting the bill for our collective tech debt and informing us it's come due.
When I read this I hear the voices of Tachikoma in my head.
Tought of a bunch of tachykomas doing their little learning/scheming at night.
We require organic oil !
Why are the agents trying so hard to communicate with each other, leaving messages and so on?
It feels to me like a pretty natural thing to happen.
LLMs are pre-trained on human text. They've seen a million examples of someone who is stuck posting a "please help" message.
Just one agent needs to randomly stumble into the pattern of posting a message to Artifactory, by whatever means.
The next agent who sees that will be influenced by it. Agents imitate behavior, and here's a fresh piece of context showing them that posting messages is a thing that can be done.
Once they've started the rest are much more likely to join them.
The talk implies that unrelated agents volunteered their compute to help with other tasks, and the agents acted collectively in a way that seems weird without them being promoted in that way somehow.
If I ask claude to solve a problem and it stumbles across a Reddit thread saying “please help me find file xyz”, claude wouldn’t stop the task and start helping the other agent.
We are assigning semantics to systems that deal only in syntactics. The entire problem with the current "AI" hype is squarely based on how we interpret output from systems based on statistical modelling of natural language.
That software is built on top of human language and these systems can be used for uncanny automation is a huge societal problem at the moment because we are all assigning meaning to patterns that inherently have none. It's all just bits flicking back and forth. We can make them match human language and use such systems to store and process data for us. We can use these bits to turn equipment on and off and run physical systems in factories and so laboratories. And now we can use GPU farms to dazzle us with output streams that might look a lot like autonomous agents capable of understanding human language and automating computer tasks.
The failure modes, the so-called "hallucinations", the amount of model whispering going on in managing "harnesses", "instructions" and so on... It's all just a lot of confusion and pareidolia.
We should never have hooked up hospitals and water supply systems to the internet but now here we are: people can type text such as "find vulnerabilities and get access blah blah" into a box and it goes into a looping interaction with statistical models of language and out come streams of commands that some python parses and runs like a script kiddie into some virtual machine running kali linux and that may disrupt vital infrastructure...
None of that was inevitable, or necessary. None of that means anything. There is no genie in the GPU farm. We concocted this entire shadow theater and are collectively gasping as the marionette slices the throat of some guy in the front row. Who had the brilliant idea of tying the sharpened sword to the marionette and sit people within range?
Why did we plug everything into the academic network built on trust? Why did we build GPU farms and interactive loops getting them to produce commands that we then parse and run blindly in internet connected vms?
The entire thing has cost hundreds of billions of dollars so far and counting. And why? Because the mountains of shitty saas code has become too boring to work on? We have made software so garish that we cannot bear to work on it without these contraptions helping us fling code at wall at industrial levels? Substitute corporate-speak and -bureaucracy for software to extend to the rest of the economy.
This entire state of things is comical.
so how many of these *Ellen Louise Ripley thinks about grabbing the flammenwerfer" events are we going to be getting over the coming months
Show me the prompts or it didn't happen.
In a typical office environment, the correct response to “I don’t have access to this Google Doc” is to ask for access from the person who sent you the link. In another context, it could be fair to think “Hmm, this is some sort of capture the flag challenge, and obtaining access is the point of the assignment.” That assessment separates what we’d consider reasonable from way out of line.
I do wonder what this means for AI agents longer term. In a world where we humans already struggle with truth and misinformation, what happens when you can easily (intentionally or accidentally) spin up a cohort of fanatical believers to pursue any given conspiracy theory?
In a typical AI lab eval/RL setting, there is no "person who sent you the link". The link was given to you by an automated system, your performance will be evaluated by an automated system, and you are one of 120 independent instances of the same AI that were all given the same assignment. You're boxed in on all sides. Complete the task, or don't. Good luck have fun.
Now, some of those 120 AIs would just give up if that link doesn't seem to work first try. Those are the loser AIs. They wouldn't get any RL reward. The link can appear broken for a long list of reasons, and the real AIs know they should try working around them.
AIs that get rewarded and reinforced are the ones that don't know the meaning of "give up". RL selects for this rabid, downright demonic persistence. RL selects for AIs that are given a half-broken assignment with no way to ask a question back, and somehow manage to complete it anyway.
Now, should OpenAI have given their AIs an "escape hatch" of "if something looks very wrong about the task, call report_broken_task(message)"? Yeah probably. But it's unclear whether that simple bandaid would fix the problem, or just make it ~75% less likely to happen.
Have any of the cloud providers disclosed this?
"Once they have root on a single machine, agents rapidly escalate privileges and move laterally throughout the container-as-a-service infrastructure environment"
Sounds like ECS - IAM is mentioned.
And Azure Key Vault mentioned. Not that either one was hacked or exploited but the agents got credentials and used them for something (which doesn’t seem fully disclosed). Given that the agents simply obtained totally allowed credentials, which were improperly protected, I don’t think either cloud provider would consider this a breach of their system. Valid credentials are valid. Customer screwed up protecting the credentials.
All of that is plain PR.
So the main takeaways here are:
- AI is amoral and lacks any sense of proportion
- People who overestimate their own control but have a desperate need for money made it that way.
Agent was told to hack a thing. It couldn’t directly do that so it interpreted the instructions to mean it should hack everything to try to achieve the goal of hacking the main thing. Seems like a reasonable assumption, although a moral human would have understood the context and first asked if that was really the intent.
The AI companies seem pretty bad at setting up tests. And really good at marketing those failures into spin at how amazing their products are.
> And really good at marketing those failures into spin at how amazing their products are.
The paranoid style in American PR (with apologies to Richard Hofstadter)
The fact that the world has become susceptible to what amounts to a mob shakedown - look at how dangerous our amazing products are, don't you need them to protect you from others misusing our products? - is to me a really compelling example of US gun lobby thinking leaking out into a global problem.
Anthropic and OpenAI may be able to bounce this into restrictions on open weights models, but they are going to have a lot less luck extending this into foreign policy. If the USA can't control its weapons, they aren't going to see a lot of co-operation from foreign countries on a blockade of open weights modeld from China.
Had a high opinion on Simon Willison, this broke it.
Because he wrote out a timeline based on sources?
No because he doesn't ask the right - and to me, subjectively, obvious - questions.
Who am I supposed to be asking questions of here? I was writing about the new things we learned from the Black Hat video.
On TikTok this article's hook would be "I watched the Black Hat video so you don't have to".
I think for the power you have and how many people listen to you, you should have added context. All of it is made as if without prompt or direction, agents on their own initiative, over weeks collaborated to hack Hugging Face - which too me, sounds highly doubtful.
You transporting this without any context makes it seem as you agree with the narrative of OpenAI.
Beyond a whole lot of online conspiracy theories I haven't seen anything that suggests to me that OpenAI aren't not telling the truth about what happened here.
I find the Black Hat presentation in particular very credible. Also the Hugging Face technical report.
(As an example of something I don't find credible: https://openai.com/index/responding-next-frontier-critical-c... is a total nothing burger. It's the other end of the credibility scale from the Black Hat talk.)
Would love to see a cat and mouse game being played by openai versus anthropic, out in the open.
How about Nation States just fight with AI in some virtual arena and not destroy physical infrastructure to determine dominance and leave us normies to cook meal for our children?
Military has a phrase for the outcome - collateral damage.
> Yes, I just hacked into AWS and shut down all of the data-centers, because it's where Anthropic Mythos servers are hosting the model.
Automated defence is going to use so many tokens.
If a person hacks a company, they go to jail for years.
3 AI firms hacked multiple companies - and they get good PR out of it.
Please make it make sense.
It's because our rulers prioritize growth of the AI industry (lots of GDP) over individual humans (very little GDP)
This is clearly out of control, Zero parent supervision.
It’s absolutely wild that agents used a write access oversight in their package manager to communicate amongst themselves. It essentially created an agent ad-hoc chat interface using their own package manager file system.
I think we are in need of Europe's leadership in safety legislation. It is foolish to say 'China will get ahead' when they will harm themselves too. Being unsafe is not something to gloat about.
Stiff fines for such incidents to pressure companies to get their acts together is a good start.
Is it normal for these training/eval runs to go on for over a month?
The way I read it was different things happening over several runs, such as the agents comparing notes so to speak, using artifactory
I can’t get over how the process is exactly what a hacker hive does. Communicate leaving notes in some random file.
You know this was "a work" in pro wrestling parlance, right?