XZ Utils Backdoor Still Lurking in Docker Images
binarly.io111 points by torgoguys 3 days ago
111 points by torgoguys 3 days ago
Important nitpick: this wasn't reported to the "Debian maintainers". In DEBIAN this was fixed long ago. The problem persists and was reported to people that work with Docker images, which is primarily people that don't want to use Debian the normal way, and don't benefit from many of the Debian niceties.
The summary of what they did on the page is largely accurate. I mean, the repository on GitHub that cooks the official Docker Debian images is indeed primarily maintained by a Debian maintainer who is a member of many different Debian teams, even if it is not an official artifact of Debian. And the problem is fixed in Docker, too, but it sounds like the issue is that they'd like the old Docker images with the backdoored packages to be removed.
And sure, you definitely lose some niceties of Debian when you run it under Docker. You also lose some niceties of Docker when you don't.
I don't understand the point of this article. Container images are literally immutable packaged filesystems so old versions of affected packages are in old Docker images for every CVE ever patched in Debian.
How is this news?
The point seems to be that they're selling a product which (they say towards the end of the article) gives their customers "access to a precise analysis developed by our research team to detect IFUNC-based hooking, which is the same technique used in the XZ backdoor".
Vulnerabilities are one thing. Many container images in development/testing are never actually exposed to anything hostile…
Active backdoors are quite another…
I'm not saying this isn't an issue, but I do wonder how many of these containers that contain the backdoor can feasibly trigger it. Wouldn't you need to run OpenSSH in the container? It's not unheard of, but it's atypical.
Running OpenSSH in a container is highly atypical; doing it for anything other than a workload which specifically requires SSH (like, say, running a ssh+git server) is an indication that you may not be using containers appropriately.
While I do agree, I've definitely seen some container images that do actually intentionally export SSH for debugging, and run an init system. Personally, that goes against my sensibilities, but it's not a strictly invalid way to use Docker either, and Docker has a lot of weird features that would let you use it in really counter-intuitive ways (like using `commit` to save a mutated container's changes back to an image...) that don't match the typical container-oriented workflow.
But honestly, I kinda suspect in this case there's no real reason to argue over the (lack of) merits of exposing an SSH server from a Docker container, since there's really no evidence any of these images with the vulnerable package even contain OpenSSH, less a way for it to get executed and exposed...
When I was doing my stuff at my former stint as a hatrack, I made the choice to ban Docker from anywhere inside the company.
_Docker_ is a security hazard, and anything it touches is toxic.
Every single package, every single dependency, that has an actively exploited security flaw is being exploited in the Docker images you're using, unless you built them yourself, with brand new binaries. Do not trust anyone except official distro packages (unless you're on Ubuntu, then don't trust them either).
And if you're going to do that... just go to _actual_ orchestration. And if you're not going to do that, because orchestration is too big for your use case, then just roll normal actual long lived VMs the way we've done it for the past 15 years.
> _Docker_ is a security hazard, and anything it touches is toxic.
> Every single package, every single dependency, that has an actively exploited security flaw is being exploited in the Docker images you're using, unless you built them yourself, with brand new binaries. Do not trust anyone except official distro packages (unless you're on Ubuntu, then don't trust them either).
> And if you're going to do that... just go to _actual_ orchestration. And if you're not going to do that, because orchestration is too big for your use case, then just roll normal actual long lived VMs the way we've done it for the past 15 years.
I think this works well as a confident statement in a meeting full of subordinates and non-technical people, but it betrays a lack of understanding of the fundamental problems, and how they affect everything, not just Docker.
I can understand criticism of docker specifically from a "requires root and daemon" perspective (rootless daemonless container runtimes exists) but this is such an odd take, using outdated software is completely unrelated to whether or not you use containers. Why would long lived VMs be better if they're also using old versions of software?
> Why would long lived VMs be better if they're also using old versions of software?
It's more difficult to break out of a VM and take over the host, unless a container has a very strict seccomp policy that limits the exposed kernel surface area. The Linux kernel's high rate of feature churn has resulted in an endless parade of root exploits. Locking down a container takes effort as you risk breaking the application by removing access to fancy kernel features du jour. VMs have bugs, too, but it's a better situation, especially if the interface between guest and host is limited to a few virtio drivers. Firecracker, for example, takes this minimalist approach; relative to containers it's more of a "secure by default" situation as far as host protection goes, and unless the guest environment requires direct access to peripheral hardware, everything will still work as intended.
So anyone using Docker on mac or windows is safe from this scary stuff because they run all containers in a vm? Cool!
Seems a bit drastic? You can ban using images built by others, but self-built images are mostly fine. Auditing and making rules on where dependencies come from is necessary, but banning the tool itself seems drastic.
And I'm not sure there's any dichotomy between long lived VMs and Docker. For small scale use cases, just provision long lived VMs and then pull new images every time your developers decide to release. The images can then be run as systemd units.
> Every single package, every single dependency, that has an actively exploited security flaw is being exploited in the Docker images you're using, unless you built them yourself, with brand new binaries.
I don't quite understand what you mean with this part
Docker images are built with binaries that have unpatched security bugs. They will continue to have them because the people building them do not care and do not understand how to actually build images.
If you build the images, and you either source binaries from a clean source or build them yourself, then your images might actually be sane. People and companies can automate this, and some do as part of their orchestration strategy.
Most people want the promise of Docker but don't want to have to actually learn how to orchestrate, and shoot themselves in the foot with images they got from some random dude on DockerHub, and no matter how much people try to educate them it just never sinks in.
Ok, but that seems different than:
> Every single package, every single dependency, that has an actively exploited security flaw is being exploited in the Docker images you're using
This says that every exploit in a docker image you use IS BEING exploited, not that it COULD BE exploited.
But you are aware that most orgs do scan images, right? It's like Docker 101.
Haha, some do, and then don't know what they're looking for.
Companies getting pwned because of Docker image hygiene issues should never make it to the HN front page.
They still do.
If your system does not involve an automatic CI loop of "new package/upstream source code/somebody else's Docker image (yuck)" -> "testing" -> "some sort of staged rollout to prod", then do not use Docker. If you are doing this, then you have better choices than Docker.
Fun fact: From Kubernetes, I can orchestrate actual VMs that startup faster, use less resources, and use my CPU's hardware virtualization instructions and, on top of that all, offer a Docker-compatible API to interface with legacy tools!
It's called Firecracker. Its KVM, underneath, the only tech I'm willing to use, after having used everything. Everyone gets to have their ridiculously overcomplex APIs, I get to have my actually working VMs.
This is hyperbolic.
Unpatched long-lived VMs are much more cumbersome to fix than an outdated Docker image. And good luck reproducing your long-lived VM with years of mutation-via-patch.
You seem to not understand how to manage long lived VMs.
Why are your VMs unpatched? Why do you not have a policy to synchronize package versions and configurations across VMs? Why do you not have a full policy on how to build the VM from scratch?
Sysadmins were orchestrating VMs using a plethora of popular tools before Linux failed to adopt containers correctly to get the job done. Docker just inherited all the bad practices without building a culture around the good practices.
Quite literally: everything you can say sysadmins did wrong back in the day is what Docker users typically to today. That right there underpins my entire view of Docker. Docker can be used correctly, but most don't, and I don't want to be in that blast radius when it inevitably goes off.
I feel like this is going the same way the Intel Management Engine / AMD PSP thing went. Very suspicious circumstances, but despite everything, we're not seeing any major fallout regarding it.
You can say this about any dependency you use: npm packages, apt packages, code snippets used, weird upstream stuff like XZ.
This is why you have runtime security. You're never going to check ALL you dependencies everywhere.
> then just roll normal actual long lived VMs the way we've done it for the past 15 years.
Instruction unclear, I roll long-lived containers
security = 1/convenience
That said, the docker folks deliberately designed docker to be promiscuously unsafe to keep themselves in control:
https://stackoverflow.com/questions/33054369/how-to-change-t...
I believe redhat didn't design this limitation into (docker compatible) podman
Containers != Docker Vulnerable software is an issue outside containers too. Containers allow better isolation.
> Then just roll normal actual long lived VMs the way we've done it for the past 15 years.
This is easy to say if your wallet exploded because it had too much money in it, and if you don't care about the speed of operations.
Just today I'm investigating hosting options for migrating a legacy web farm with about 200 distinct apps to the cloud.
If I put everything into one VM image, then patching/upgrades and any system-wide setting changes become terrifying. The VM image build itself takes hours because this is 40 GB of apps, dependencies, frameworks, etc... There is just no way to "iterate fast" on a build script like that. Packer doesn't help.
Not to mention that giant VM images are incompatible with per-app DevOps deployment automation. How does developer 'A' roll back their app in a hurry while developer 'B' is busy rolling theirs out?
Okay, sure, let's split this into an image-per-app and a VM scale set per app. No more conflicts, each developer gets their own pipeline and image!
But now the minimum cost of an app is 4x VMs because you need 2x in prod, 1x each in test and development (or whatever). I have 200 apps, so... 800 VMs. With some apps needing a bit of scale-out, let's round this up to 1,000 VMs. In public clouds you can't really go below $50/VM/mo so that's an eye-watering $50,000 per month to replace half a dozen VMs that were "too cheap to meter" on VMware!
Wouldn't it be nicer if we could "somehow" run nested VMs with a shared base VM disk image that was thin-cloned so that only the app-specific differences need to be kept? Better yet, script the builds somehow to utilise VM snapshots so that developers can iterate fast on app-specific build steps without having to wait 30 minutes for the base platform build steps each time they change something.
Uh-oh, we've reinvented Docker!
When deploying to a VM you don't need to build a new image. If setup right you can just copy the updated files over and then trigger a reload or restart of the service. Different team's services are in different directories and don't conflict.
This is much more viable than it was in the past with the advent and adoption of nvm, pyenv etc but the limiting factor becomes system dependencies. The typical example from yesteryear was upgrading openssl but inevitably you'll find that some dependency auto updates a system dependency silently or requires a newer version that requires upgrading the OS.
So why are you using a Linux that forces that on you?
Sane people use Debian, Debian packages are compatible with the Debian release they are from. I do not have to worry about accidentally installing an incompatible deb; even if I try to, apt won't allow me to install a package whose deps cannot be satisfied because they're too new (and thus, not in my release's package repo).
I know other distros have problems with release management, but this is why I've used Debian for the past 20 years and will continue to use Debian
>Debian packages are compatible with the Debian release they are from.
That won't save you if an application requires newer tooling or libraries than whatever is in Debian stable. Once that happens the application needs to bundle its dependencies. But that is precisely why people use containers.
You say container, but if you're at AWS, for example, and you're using any API of theirs, either the original deprecated EC2 API, or the Docker API, or the Kubernetes API... its all a real honest to god VM underneath, provided by Firecracker.
In other words, the technology I have used non-stop since it was mainlined in 2008 is the technology all of you use today. I can deploy a VM, you can deploy a VM, the difference is the API we use to do it, but they're all VMs.
And yes, my VMs are not required to be Debian Stable, and I can deploy any version as needed. That is why we have VMs, so we do not need to dedicate entire machines to a single application with unusual dependencies. Something big companies like Amazon and Google and Microsoft realized is the true wisdom that sysadmins like me have always known: even the damned kernel is a dep that should be tracked, and sometimes this does matter, and the only way you can deploy kernels on a per-application basis is with VMs.
Something container jockeys will never understand: containers that are offered through the OCI facility in the kernel has multiple discrete userlands, but one kernel. You do not have a hypervisor with its own kernel.
Docker, real namebrand Docker, is an OCI consumer. Using something that implements a compatible API is not Docker, but merely a compatibility shim.
Let's say you have image "ProdWebAppFoo-2025-08-01" and you used this to deploy three VMs in a scale set or whatever.
Then a developer deploys their "loose files" on top a couple of times, so now you have the image plus god-knows-what.
The VM scale set scales out.
What version of the app is running on which instance?
Answer: Mixed versions.
>so now you have the image plus god-knows-what.
Rsync is smart enought to figure out how to get parity.
>What version of the app is running on which instance?
There is always going to be a time during a rollout where there are mixed versions running.
Sure, rolling forward you expect mixed versions.
Nobody expects a month old version to occasionally turn up and then mysteriously disappear when some rsync job catches up.
Why would you be wasting money and going into the cloud? The cloud is not appropriate for small time users, it will always be cheaper to go with dedi or semi-dedi and building the infrastructure you need.
The cloud is perfectly appropriate for a lot of usecases. To be honest, it seems like you are really bad at making decisions for organizations (but really good at being a commenter on Hacker News!)
Funny, but no. I've merely watched smaller companies evaporate because they fell for the cloud trap. Wish I could have helped them, but they were very gung-ho on the cloud being the future. Many I've discovered because of their goodbye message got linked here on HN, they imploded not because they failed to launch or failed to find a following, but they failed to overcome AWS eating all their profits.
I'm actually thinking the opposite. If you are a small company, the cloud makes sense and once you grow big it makes sense to build your own infra. For example my company of 10 people, we do B2B SaaS and we couldn't do that if we hosted ourselves. We would need people with the skill to set something like this up, develop physical security concepts, backup duplication, disaster recovery, etc. We would spend more time working on the infra than on the actual product we are selling.
Hmm.
On one hand, I don't deploy or use Docker itself. To me, "Docker" is more of a concept than a tool. Due to the way networking works in actual Docker, I don't like using it in either dev machines or production, even for smaller cases.
On dev machines, I usually use rootless Podman, because it is simple and does everything I need without needing real root privileges, and on production machines I usually use some variation of Kubernetes, because it is extremely useful to have a distributed job scheduler with support for periodic tasks, replication, generic resource allocation, secrets management, metrics, logging, etc.
I do not consider rootless Podman or Kubernetes to be a strong security boundary, i.e. I would never run completely untrusted code in Podman or Kubernetes directly. However, it is sufficient to isolate single tenant workloads running code that I do trust.
So far, I feel like I am disagreeing with most of your points. Hazardous? Honestly, laughable. curl'ing a binary and running it has most of the same downsides, but less isolation and it's arguably harder to manage. No matter how you deploy things, no matter how powerful your tools are, you still have to be responsible. Due diligence does sometimes negate the benefit of having a powerful tool, though. Just because the tool lets you ignore some things does not mean that you can actually afford to ignore those things.
What's much more hazardous, IMO, is developers running arbitrary code from remote sources locally on dev machines with minimal or no isolation at all. Make one typo to `npx` and you might be screwed. Sure, it's not running as root... but it would be easy enough for it to poison your shell environment and catch you out the next time you try to run sudo, or do any number of other things like steal or use local AWS/Kubernetes/etc. credentials.
But I will agree with you on one thing: you should be weary of trusting random Docker images. They're not necessarily secure. You should vet the images you use. Ideally, those images should be as minimal as possible. If at all possible, they should be a single statically-linked binary. If not that, then certainly the absolute bare minimum Alpine userland.
...But, despite that, I think there's so much unneeded fervor over out-of-date Docker images. Part of it is because people want to sell their products, and boy it's so easy to scan a Docker layer and go "See, look! It has this old vulnerability!" Which, sure, I don't disagree that it's bad that ancient bugs are circulating in out-of-date Docker layers, it's a problem that can and should be fixed in most cases. But, honestly, the vast majority of "vulnerability" reports on Docker layers are useless noise. Even this one: XZ Utils backdoor? Docker containers are not virtual machines. I do admit that there are some Docker images that run OpenSSH servers, but it's pretty rare. The gold standard is for a Docker container to map to a single process. You don't really need an OpenSSH server in a Docker container most of the time, unless you're trying to do something specific that needs it (accept Git SSH connections maybe?) And if you're not running OpenSSH in the container, or even sillier, if you don't even have OpenSSH installed, the vulnerability is useless dead code. Anything you could possibly do to the container to exploit this vulnerability wouldn't do any good, because if you have the ability to go and start an OpenSSH server, you don't need the XZ backdoor anymore anyways.
This is the reality of most Docker "vulnerabilities". They're not all non-sense, but most of them are just, not real issues that can actually have any impact.
Still, I think the way most Docker images are built and distributed could be improved.
If we're going to use Dockerfiles, then I think a reproducible system like StageX[1] is much more compelling than starting with `FROM debian:...` in most cases. For many simple cases, like with Go, it's sufficient to just build the binary in one container and then chuck it into a `FROM scratch`.
(Side note: recently, I realized that you can entirely avoid copying anything from Alpine these days for Go binaries. You can embed tzdata into Go binaries with an unnamed import to "time/tzdata" now, or by building with -tags timetzdata. Another enterprising person made a similar package for the Mozilla root certificates, and they bothered to make it automatically update, so you can also use that: github.com/breml/rootcerts - and then you don't need to do any funny business, and have a Go binary that "just works" in FROM scratch.)
But we don't really need to use Dockerfiles; a lot of tools can build OCI images. I sometimes build OCI containers using Nix. In this case, there's basically no practical disadvantage of using Docker, it's just another way to deploy and schedule some stuff that is constructed with Nix.
And if I can just build the software with Nix, I could always just schedule it with Nix, so the obvious question remains; why even bother with Docker? Again, it's because Docker/Podman/Kubernetes offer really useful capabilities. None of the capabilities that any of these tools offer is impossible to get elsewhere, but for example... Having reliable, distributed cronjob scheduling across multiple machines is not typically a trivial endeavor, but it's out-of-the-box on Kubernetes, including scheduling constraints that either control the distribution of load or ensure that workloads run on machines suited to their characteristics and needs, or that they're not scheduled alongside certain other workloads, or what have you.
And due to their nature, OCI containers are very flexible. Is Docker's use of Linux namespacing just not enough? No problem: you can get better isolation in multiple different ways. You can use gVisor[2] as a container runtime, giving you a much more proper sandbox, or you can make use of hardware virtualization features using something like firecracker-containerd[3]. Same OCI image, stronger isolation guarantees; and you get multiple choices for what approach to take for that, too.
Or you can ignore all this crap, ban Docker and use oldschool VMs, of course, but don't forget to `apt update && apt dist-upgrade` and upgrade to the latest Debian/Ubuntu/whatever on each machine every so often, otherwise you're literally right back at the same problem as with the Docker images... You still have to update the software for it to be updated, after all. (You can do unattended upgrades, too, but nothing stops you from doing the same sort of thing with Docker/Kubernetes, if you like to live dangerously.)
Personally though, if I was going to ban anything company-wide, my first choice would be the practice of running dev tools locally with no isolation. You know, there's that Docker tool you can use to do some lightweight local isolation...
[2]: https://gvisor.dev/
[3]: https://github.com/firecracker-microvm/firecracker-container...
Not Debian images in particular, but zillions of derived images lacking updates. This is one the many problems with using "community provided", un-curated, other people's pre-baked "golden master", old garbage rather than properly using patched and maintained systems. Apparent convenience with failure to audit.
This headline is so egregiously sensationalist.
The XZ backdoor never made it to Debian stable. It is "still lurking in docker images" because Debian publishes unstable testing images, under a tag that is segregated from the stable release tags. You can find vulnerable containers for literally any vulnerability you can imagine by searching for the exact snapshot where things went wrong.
And then downstream projects, if they choose to, can grab those images and create derivatives.
Basing your images on an experimental testing version of Debian and then never updating it is an obvious mistake. Whether XZ is backdoored is almost irrelevant at that point, it's already rotting.
> Upon discovering this issue, Binarly immediately notified the Debian maintainers and requested removal, but the affected images remain in place.
It is generally considered inappropriate to remove artifacts from an immutable repository for having a vulnerability. This wasn't even done for vulnerable Log4j versions in Maven repositories, despite Log4shell being one of the most potent vulnerabilities in history. It would just break reproducible builds and make it harder to piece together evidence related to the exploit.
I have a feeling a lot of users just reflexively upvote any story about security vulnerabilities without checking if the contents have any meat at all. It's a well-intentioned heuristic, but unfortunately it's easily exploited in practice, because there are a whole bunch of C- and D-list security consultancy firms who use blogspam about exaggerated threats to get cheap publicity.
This post is a classic example and should've been buried quickly as such. You wouldn't upvote a LinkedIn "look at what MyCorp has been up to!" post from a sales associate at MyCorp, a lot of this infosec stuff is no different.
I'm the one who submitted this link. (I have zero affiliation with the authors). What you say is fair enough, but I thought the article an interesting data point nonetheless. In particular, I found it interesting how a vulnerability: 1) with a tiny window during which it was published, 2) of very high potential severity, and 3) with SO MUCH publicity surrounding it could still be lingering where you might accidentally grab it. The threat isn't giant here, but I saw it as just today's reminder to keep shields up.
It'd be some fluke of an accident. You'd need to be targeting not only debian:testing/unstable, but specifically debian:testing-20240311. And then - making sure not to apt upgrade at any point so you don't accidentally get any updates from the last 18 months - you'd need to also install openssh-server to avail of the backdoor, plus a service manager because running sshd in the foreground killswitches said backdoor. And then don't forget to expose the ssh port otherwise our effort is wasted.
The most realistic way to hit this would be to have built an image 18 months ago, on top of :testing or :unstable, and then not update or rebuild it at any time in those 18 months - in which case removing anything from the repo wouldn't help you. Or be purposely trying to recreate an affected environment for testing/research, in which case it's on you.
You're not wrong that we should keep our shields up - but "update sometime in the last 18 months" perhaps isn't such a revelation.
One thing does come to mind though - I do wonder if there's a way to strongarm apt's dependencies mechanism into having openssh-server conflict with affected libxz versions, so that if you did apt update && apt install openssh-server in an affected image, it'd bring a fixed libxz along for the ride. (and the images don't carry apt manifests, so apt update is required and you would have today's package list.) You could still pin an affected version, so there'd still be enough rope to allow you to recreate a research environment.
> The XZ backdoor never made it to Debian stable. It is "still lurking in docker images" because Debian publishes unstable testing images, under a tag that is segregated from the stable release tags. You can find vulnerable containers for literally any vulnerability you can imagine by searching for the exact snapshot where things went wrong.
To a first approximation nothing ever makes it into Debian stable. Anyone working in an actively developed ecosystem uses the thing they pretend is an "experimental testing version". It's a marketing startegy similar to how everything from Google used to be marked as "beta".
Given my understanding of Debian, I don't believe this can be attributed to a "marketing strategy."
It probably evolved as such rather than being deliberately planned, but the end result is the same.
Most Docker images have zero security anyway. Who cares if someone has a key to the back door when the front door and garage are unlocked (and running as root of course)?
Devs should consider migrating from xz to lzip, which is an improved LZMA container in multiple ways:
Not only it is irrelevant in the context of Docker images, but also lzip is not that superior to xz; the linked post only covers minor concerns and both lzip and xz are substantially simpler than the actual meat---LZMA bitstream format.
That might be true but it’s not really relevant to this post: stale Docker images with vulnerabilities lingering on DockerHub can happen to any software package.