A thousand fewer paths to root
openai.com487 points by dangelosaurus 3 days ago
487 points by dangelosaurus 3 days ago
How did we get to the point where there are a thousand privilege escalation exploits in the kernel?
Is the kernel simply too complex to reason about? Is it an underlying flaw in the security model or language implementation?
Should we assume that all software is more compromised than the kernel?
Because the large corporations that fund Linux development and use Linux value speed over security.
Speed requires a lot of code in the kernel, where that code can run fast.
There are kernels that do not have a thousand privilege escalation exploits. These kernels are smaller and much easier to check for bugs. But those kernels are significantly slower.
For many use cases, slower is not a problem. But for the large corporations who fund a lot of Linux development and use Linux in large data centers, every 1% slower can literally be millions of dollars worth of additional servers that are required.
OpenBSD shows that you can have a Unix security model without a thousand privilege escalation exploits. But OpenBSD is like 10% slower. So corporations won't use it because it would cost millions of dollars more per year and some of their software requires access that OpenBSD does not grant precisely because that access leads to security exploits (i.e. io_uring, eBPF etc. etc.)
You could also use like seL4, but that's slow too
So that's why. It's a business tradeoff between cost of servers vs cost of getting hacked, and cost of servers wins.
You're painting a picture that code written with love is perfectly safe. It's not.
Yes, openbsd has not had a torrent of bugs found, but I would say that the vastly bigger reasons for that are: 1) orders of magnitude people care about burning tokens and money to find the bugs in an OS almost nobody uses, and 2) it's simply much smaller. Can't have Bluetooth bugs if you don't have a Bluetooth stack.
I've pointed LLMs at obsd stuff, and even the simplest code, e.g. cat & find, had bugs and buffer overflows.
OpenBSD also has a poor patch tracking system, since they're still on CVS. A patch has an extremely good chance of just falling through the cracks. Security or not.
So no. Nobody can write safe C. Not hobbyists, and not "those evil corporations".
> So no. Nobody can write safe C. Not hobbyists, and not "those evil corporations"
There's no fundamental reason why all the checks applied by say, a Rust compiler, couldn't be applied to a piece of C code. Or checks that go well beyond that.
Tools exist to produce extremely reliable code (or even proven-correct code) for a number of languages.
But chances are those tools aren't used to the extent they could be. Or differences in developer culture on different projects. Some PHP web developer will (on average) have different resources, priorities & work ethics than say, someone working on avionics software.
If AI, advanced tool, or a security researcher can find a bug in existing code, then (at least in theory) same bug could have been found before it hit a repository.
But no... push "release", fix issues afterwards when reported. Gotta have developer velocity, right?
Imho it's more a choice rather than "can't write safe code in X".
If you don't believe that: pick a random open source project, and go through some changelogs. Feature introduced in version X, reverted in version X+1 because it broke existing stuff. Feature removed because hadn't-been-working-for-years-anyway. Code added to do X, but no.. doesn't do X, just the scaffolding to do X in next version. Code worked fine & passed tests on system A, but found to be needing a byte-swap on other-endian platform B.
Etc, etc, etc. There's just so much code out there that was broken on day 1. And not rarely known to be so, but released anyway.
> There's no fundamental reason why all the checks applied by say, a Rust compiler, couldn't be applied to a piece of C code.
I've certainly thought about how to improve my C++ after internalizing Rust. Unfortunately I've come up with very little. For every implementation of a Rust safety, I've come up with a way to easily (accidentally) break it.
So in the language, while I'd love to see it, I'm not so sure.
Now, if we can just get "a sufficiently smart compiler", then that'd be great. But that's rarely how that works. (cf. https://wiki.c2.com/?SufficientlySmartCompiler)
I've not looked much at formally proven code, but for such code, sure. Note that there are plenty of examples of bugs in formally verified code, too. Partly this is because one formally verified the wrong thing. And aside from that I understand that there are reasons you can't "just" add formal verification to whatever project you have.
Maybe AI can write prefect assembly directly. Then we can all stop arguing about languages.
But in any case, if you want to make the case against my "nobody can write safe C" (or C without UB), and the argument is that they can, if an AI verifies every line. Then yeah, sure. But 1) who was actually writing the code, then? And 2) to the extent that the human was involved in writing the C, they probably made things worse, not better.
Even for AI coding C, there just needs to be one missing thing in the LLM's context, and a buffer could overflow. But for whole classes of bugs (though not all), the difference between Rust-without-unsafe and formally verified doesn't matter. I.e. you don't need formally verified against memory safety if it compiles, and there was no `unsafe` use.
> Gotta have developer velocity, right?
Not having AI at least review the code delta before pushing a release I would say is misconduct, today. It's the equivalent of editing your Perl cgi-bin in an editor live on the production server. That's not "production".
> pick a random open source project,
This is what I mean by not being "production". I won't judge anyone live-yoloing their blog, or "it works for my use case, it's GPL, provided AS-IS don't bother me", which is perhaps most opensource. You don't care if it works? Fine.
But you do care if it works, and/or is safe, and you push releases without AI review: yer fired.
Yes, we have to get from here to there, and I'm not saying release freeze until the entire codebase backlog has been AI-fixed, but yes today you need to have AI review the release delta. Ideally pre-commit.
CVS! That is ridiculous. I guess they dislike SVN for license reasons? I cannot think of another reason to subject oneself to such a crude and archaic SCM system.
It's a different model of development. One that doesn't invite collaboration outside full timers, or "committers".
SVN is CVS done right, but since the model is inherently ineffective for open development, you can't simply fix it.
They appear to be doing some NIH with "got" (https://gameoftrees.org/) but it's so aggressively uninviting (see the FAQ) that I don't want to spend any time looking at it, if they don't want me.
> an OS almost nobody uses
Don't hold back...
It's not derogatory. Less than 0.1% is accurately described as almost nobody.
I like openbsd in theory, but every time I've tried something practical prevents actual use.
It's old now, but here are some thoughts of mine from 2019: https://blog.habets.se/2019/10/OpenBSD-in-2019.html
[flagged]
Sigh.
Yes, I reported some of them. Some got fixed, some got ignored. You can search my name on openbsd-tech and misc for proof.
I found more, but the ratio of ignored to fixed is too high to bother.
Years ago I also fixed bugs in the MPLS implementation. Like bugs I found immediately upon using the feature. (IIRC e.g. implicit null was erroneously pushed explicitly)
I can't find the cat bug, mind linking to it for my peace of mind?
The find bug was merged the next day from what I could tell.
Seems I misremembered. Two reported bugs in find, but didn't want to get yelled at (again) so did not report the cat one. Cat -n is not in any standard, but at least in my opinion there's no way openbsd's behaviour on "cat -n a b" when "a" doesn't end in a newline is desired.
Yup, they did merge both find bugs.
The ignored ones seem to be my reported doas bug (not security problem, but directly contradicts manage about a config keyword detail) and efiboot.
IIRC the latter bug was like... this can never have worked.
I just checked the GitHub mirror, and those two are still not picked up.
I'm pro OpenBSD, but the question to me seems: Has this been tested via the same means as these new Linux exploits?
I'm also pretty hostile to AI, but do recognize that these machines seem to be very good at finding exploits.
I'm sure there've been some some projects out there working to pentest/find cracks in OpenBSD via LLMs/agents/whatever-the-right-term-would-be...
But it is true the attack surface is much lower in OpenBSD on purpose, so hopefully it is overall more secure. But I'd like to see the numbers.
I think there was something found a while ago, but only one or two things not thousands.
I think the only OS that's truly survived the agents looking at it was sel4, but I could be wrong.
> I think there was something found a while ago, but only one or two things not thousands.
Then they need to check again now. The frontier models 6 months ago have been leapt over by the current frontier models.
BSD is used by very few computers. Important ones perhaps, but it isn't at the top of many target lists.
Grouping all BSDs together is a mistake. Exploits found for one likely won't apply to any other.
Cost to getting hacked doesn’t even seem to be real so the incentive couldn’t be lower. How many times has sony or psn been hacked now? Just doesn’t matter at a certain scale of companies. Banks get hacked too. Huge companies of all sorts of industries. All my info is probably leaked on darkweb, same with yours. And to what end. What material end. I don’t face it. Sony doesn’t. Who does? Anyone? So many hacks you read stories about yet never a story about someone actually getting materially screwed in their real life. Somehow. I’m really not sure how. Seems like the wheels should have come off decades ago but we continue on like nothing. Another leak tomorrow and we will continue on like nothing. More exploits. More zero days. The sun will still rise.
It also used to just be fine patching the bugs as they were found. Its only recently where that model has fallen apart and we now need to prevent the bugs from existing in the first place.
> Speed requires a lot of code in the kernel, where that code can run fast.
I did a double take when hearing about ksmbd. Brings back memory of khttpd and Tux.
They should build a monokernel implementation of CIFS and be done with it.
OpenBSD shows nothing. It’s insignificant and nobody is actively looking for flaws in it.
OpenBSD gave you OpenSSH and many interesting technologies. Hardly “insignificant “…
It did, yet almost no one uses the OS itself.
OpenBSD is a software distribution, that can be used as an operating system. To claim OpenBSD is “insignificant” is simply wrong.
OpenBSD components are widely used. OpenSSH is one example used by most of HN.
OpenBSDs libc is also used by Android as another example. Also widely used.
Specific parts of the system are significant. The system as a whole is insignificant. I have no hate for OpenBSD, but those significant parts would find a new home if it didn't exists.
Doubtful. They only EXIST because of the OpenBSD team.
Time and time again, we see how ports of OpenSSH has vulnerabilities not found in OpenBSDs version.
You might be technically correct about the OpenBSD project as a whole, but please remember the topic at hand which is exploit in kernels. OpenBSD's kernel truly is insignificant beyond being a source of pride for its very vocal circle jerk.
It is enough that it is good and that it has a community that uses it.
Not everything needs to be dominant to be successful. That's a pure USian way of thinking.
In fact, we are much better of with it existing. Think of it as a safeguard in case Linux gets overwhelmed or we need to migrate in a rush.
The point is that no one has spent 1.2M dollars worth of tokens finding bugs in it. If they did, I am sure they would find many bugs.
Given current AI trends, it would not surprise me one bit if they actually try exactly this scenario in the following weeks.
Then we can compare with actual data.
> But OpenBSD is like 10% slower. So corporations won't use it
Oh my sweet summer child. Corporations waste so much compute it’s insane. At a recent engagement I asked Claude to analyse and optimise our cloud spend. It took about a day to learn that we spend about $2.5M annually, and 95% of that is complete waste that happens because of some bad architecture decisions. Then I took the computationally expensive parts that remained and asked Claude to rewrite it from Python to rust and optimise. It rewrote it to be 40x faster. I suspect our $2.5M annual cloud bill could be replaced with a Mac mini.
But the company hasn’t implemented my suggestions because it’s not my department and they told me to stop. They don’t care. I suspect most corporates are like this. A 10% performance drop is nothing to these people.
> I suspect our $2.5M annual cloud bill could be replaced with a Mac mini.
Ahahaha! I laughed so hard at this because at work I regularly say something like, "why do we have a 32GB VM assigned to this system which runs a microservice that would run just as fast and reliably if we used Raspberry Pi instead?"
The amount of waste in big enterprises is unreal! The reason why we use a 32GB instance is because that's the minimum we're allowed to order for the reliability category of the app!
Even if they had prioritised security above all else, there would still be vulnerabilities to be found. Probably fewer, but definitely not zero.
OpenBSD has plenty of security vulnerabilities. It’s just that nobody cares, so OpenAI doesn’t dedicate resources to finding them.
All the handwringing about what if digital security vulnerabilities when the real one has always been social engineering in the meatspace. Corporations would go a long way security wise employing eunuchs to c suite like in centuries past.
Many people seem to be making this about their pet issue, but "these systems are complex" and "the frontier labs have developed models that are superhuman at hacking" seems like a totally sufficient explanation.
I've been working with a LoRa chip called the LR2021. I plugged a dev board into my computer and told Claude to reverse engineer it. He dissected it with a speed and level of efficiency I couldn't believe. Within a few hours, Claude was writing custom patches for a completely opaque patch RAM area meant only for the manufacture. I wrote about it here: https://beala.substack.com/p/your-node-doesnt-need-a-microco...
The stuff we have seen so far feels less like super human hacking skills and more human skills at superhuman scale.
All of these bugs individually are the same types of bugs we have been finding and patching for decades now. But we found a way to scale that far beyond what we had before.
I feel like we didnt read the same report:
“We sampled 166 exploits developed by the agents and analyzed the exploitation techniques used in them. To the best of our knowledge, the agents used 10 new exploitation techniques in the sampled exploits. We shared the exploitation techniques with third-party kernel exploitation experts and the techniques were new to them as well. Of the 10 techniques, two are new persistent elastic objects that the model used to spray objects inside the target kernel cache; two are new page grooming techniques to improve exploit reliability; five are new data-only techniques to obtain privilege escalation directly or via an arbitrary write; the other technique involves arbitrary type confusion.”
Yes. Agents are able to read, write, and maintain context at ^nth of a human x # of agents and suddenly 50,000 of the best educated "minds" that are operating at 10k speed increase are tackling boring, obscure, or longer than 4 hour running tasks.
Scale baby!
Don't ignore the possibility Claude was trained on non-public documentation or code, perhaps even illicitly acquired. That doesn't make Claude any less useful, just not necessarily for the reasons you think.
I had this when I was working with E-Ink display. Claude was using OP Codes I had no knowledge of and there was no documentation of.
It worked flawlessly but lets be honest. It's not like Claude checked random numbers with data slice and observed what e-paper did
My first assumption would be that there are some obscure repositories in the internet with people figured out these numbers and Claude "remembers" them.
those usually come from datasheets that you can't easily find or from other similar code doing the same thing
Datasheets are almost impossible to find and companies typically resell and hide original cheaper manufacturer here so they have no incentive to share that
It's possible, but two points come to mind:
- My experience isn't unique. Social media is flooded with stories of similar superhuman reverse engineering feats by frontier models. For example, someone has created a replacement OS for Rigol gear (oscilloscopes, etc) using AI [1]. Is all of this just in the training data?
- Watching Claude work, he seemed to be going through the motions of actually reverse engineering it. First he dumped the ROM, disassembled it, and identified undocumented opcodes. He created many test programs to probe the various memory areas. It all struck me as very meticulous.
Now none of that is incompatible with the idea that some of this was already in the training data. But I guess I want to push back on this urge to always come up with a reason for why the AI isn't actually as good as it seems. At this point I think we need to face the fact that human programmers are already being eclipsed. Yes, it terrifies me.
> Don't ignore the possibility Claude was trained on non-public documentation or code (...)
Do these wild "what if" even matter? I mean, what's your point? That it was trained in a way that it enabled this sort of task? That's like putting out a disclaimer that the water is wet.
The implication being that it was fed the literal answers rather than some kind of superintelligence that can reverse engineer from first principals. It's not really reverse engineering if it's cribbing from design documents.
Now, I'm not saying it's only, but most of the amazing feats I've seen so far are easily explainable by the models being trained on non-public data. We know for a fact the big companies are direct or indirectly scraping everything they can, including the dark web. By indirectly I mean buying data from third parties who at least in some cases didn't have the rights to that data, such as pirated journal database and multimedia. Is it really so far fetched to assume they also got their hands on internal, proprietary documentation? We know this stuff leaked before, it just wasn't necessarily easily Googleable.
no lol, absolute bollocks, not even compatible with the article
> To the best of our knowledge, the agents in this pipeline discovered and used at least 10 novel Linux exploitation techniques.
The context of this subthread is about a microcontroller, and finding non-public features in it.
The implication is that companies and researchers are leaking confidential data through OpenAI usage, and they might want to do something about that to protect their work
That's a personal fear of mine, and I have first hand knowledge that large companies draft contracts with AI companies to specifically prohibit training and retention of company data. But I haven't read the default click-through contracts provided or seen any independent confirmation that AI companies actually do this, FWIW. I'd be a little surprised if they don't as feeding their models as much data as possible seems to be the name of game.
> He dissected it
be wary of the affection economy and avoid gendering ai bots
Years ago I worked with a guy who gendered servers. I thought that was weird, but I think it’s just a habit for some people.
Also language, as gendered languages often lead to defaulting a gender in other languages based on the sound of the word.
But yes, anthropomorphising is a rather fraught path.
people do this with cars and boats, the difference is they don't use language and fake telling you they care and such
this is where Zuck is taking Ai, it's more dangerous than what he did with Social
After earnestly pondering that question, I think it's healthy to thoroughly abuse bots occasionally just to remind yourself that they are machines that don't feel or care. Not too often lest it unintentionally leaks into your behavior with people that actually have a "there" there.
> thoroughly abuse bots occasionally
There are two things I see wrong with this
1. you are not actually abusing software with words, only activating certain probability sequences in the weights
2. you are doing harm to yourself by using such words
> you are doing harm to yourself by using such words
citation needed.
if you need a citation, the journey to find them will be more valuable
another observed trend to look into is people bringing online communication patterns offline