The Polyglot NixOS
x86.lol119 points by todsacerdoti 4 days ago
119 points by todsacerdoti 4 days ago
I have to say that that "surprisingly simple" thing is happening more and more for me on NixOS as well.
Recently a customer wanted me to use Fedora (I never visited the RPM side of the world before), and after truly the worst installer I've ever used, the actually system was nice. I do like Cockpit.
But then I needed to install an initrd that would me unlock the full disk encryption via SSH (it's a remote headless box). It took me half a day and a forum post to get it to work. I wrote a full page of notes for next time. Then the Firewall: I hit a bug (plus user error) which left me wrestling with a non-existent Tailscale interface for a while (it warns you for non-existing interfaces, but not with only a case mismatch, it then lets you do everything as if the interface exists), but after some hours I was done setting the zones, another page of notes and commands to enter to get to the desired state.
These configurations are both 1 or 2 lines in a NixOS config file. And that "work" is now done for all my NixOS servers.
You could argue that NixOS hides a lot of complexity, but so do Dracut and Firewalld of course. Nix is difficult, it's a high level abstraction. But it also just a bunch of key-values, and write-once, deploy everywhere.
>> You could argue that NixOS hides a lot of complexity
They both have the same complexity in that scenario. Underneath it's very comparable configuration for both but Nixos provides an easy abstraction for that specific case.
If you can stay on the happy path with nixos then it's pretty lovely. I've even adopted nix-darwin for my mac's too.
I'd still deploy Redhat/Fedora over nixos on anything revenue generating though. The problem is when you have to come off the happy path in nixos and now you're debugging some interestingly written c++ code that evaluates a language that has a derivation expressing what you wanted done. Contrast with the redhat situation, it's simpler but less convenient in the general case.
> The problem is when you have to come off the happy path in nixos and now you're debugging some interestingly written c++ code that evaluates a language that has a derivation expressing what you wanted done.
I agree that Nix or NixOS are risky tools.
But it's not a problem that the `nix` program is written in C++.
A lot of the friction comes from the tension between NixOS's idiosyncratic design & its constraints, against the often 'dirty' way software is practically written.
For example, roughly, Nix's ideal software follows `./configure && make && make install`, where nix can then symlink the dependencies & maintain these in a read-only store. -- Whereas, say, Python wheels break this (because the precompiled binaries assume shared libraries are available globally).
When you run into friction with NixOS, you may need to understand things with a depth and breadth which aren't required with more common Linux distributions. (Including e.g. maybe needing to understand the rather large and obscure nixpkgs).
I'm going to mildly disagree here.
NixOS is an improvement over RH et. al. even if you stray a small amount from the happy path; to use the GP example, if they couldn't do what they needed with the nixos LUKS options, they can click on a link from a related option definition and end up with https://github.com/NixOS/nixpkgs/blob/nixos-25.11/nixos/modu... which will describe much of how luks is configured. Often a small change there can widen the happy path, and with a PR, it can be made upstream in unstable relatively quickly and a release in the next several months.
Do you have examples of times you've had to dig into the nix language like that, at least at a high level?
Just curious. I haven't run into anything like that myself.
Using nix for three months now, the main "pain" I run into is that I want app config files to remain writable by the app, but home-manager ethos is understandably against that -- and this is what you want on servers.
So I've had to vibe-code my own HM module for claude code, keepassxc, cursor, etc. and use activation to merge my nix settings into those files if they exist. That way when I rebuild, my nix config can assert a subset of the config I care about without locking the app out of writing to config -- and this makes more sense for personal desktop computing.
Though I put pain in quotes because it's still 10x better than what I was doing before, and an LLM can do it just fine.
I do agree with that. But as a result I've gotten pretty good at whipping up podman containers with various bases to solve such issues.
The difference is, you can create proper abstraction modules that put everything together, from dependencies, to config files, firewall rules etc and have nice options for your abstraction.
No other system provides this in that sane way. I used countless configuration systems, from custom bash hacks, ansible, chef, puppet, salt - I have seen a lot.
Nix is just on another level. Never going back
> The difference is, you can create proper abstraction modules that put everything together, from dependencies, to config files, firewall rules etc and have nice options for your abstraction.
Yes, this is an understated benefit.
The declarative interface is nice.
That NixOS configs are modular allows you to create those abstractions. -- In the same sense terraform modules are "infrastructure as code", NixOS modules are "system configuration as code".
My problem with those abstractions is:
1. That the error messages for them i find to be awful. They really need to take inspiration from Rust to help users know exactly where the API error is happening relative to your code. I've spent quite a lot of time toggling things on and off to try and find which part of code, or which package, is causing a specific failure.
2. I've not found the APIs that the abstractions provide to be stable in the same way my Rust crates/etc are. Something can randomly break between updates and i'm not sure what caused it. Takes me quite a while of digging through source to find the root cause, and digging through Nix source i find extra painful.
All around i hate this side of Nix. A lot. However it's enough of "another level" that i stay with it and don't switch away.. I use it for my Linux machines and my Mac laptop. It makes so many things so easy, the majority of the time.. but when something goes wrong.. it's super painful.
As someone who up until recently would have agreed with you, both of these are fundamentally familiarity issues. Nix's error messages really aren't that bad, but chances are there's exactly one line out of 200 that tells you what's wrong. Learning to read stack traces for a new language is part of learning that language.
I've not had issues with Nix APIs, at least not Nixpkgs or the language builtins. When something does break for me, it's usually some random JavaScript package that had some external dependency change. Nixpkgs is pretty well organized and I find navigating it not that hard once you read the packaging guidelines. find / fzf / ripgep / etc. are all great at this, as file and folder names are critical to the organization of nixpkgs.
The big turning point for me was trying to build and package a non-trivial application and build a NixOS module for it.
Package errors aren't that bad, but NixOS config errors are a nightmare because they are often a value that was computed from a value that was computed from what you wrote triggering a missing field, class mismatch, or infinite recursion in very generic framework code.
Agreed on 1, thats part of the cost you pay when choosing Nix.
Disagree somewhat on 2. Yes, there are frequently small breaking changes, but the massive upside of nix is I can either just rollback or pin a specific package at a specific version and move on.
Yeah once you've used Nix[OS] and home-manager, it's hard to go back to apt or brew or the dreaded "sudo make install" without feeling like you need to have a shower afterwards. And I was a loyal Ubuntu person for like 15 years. It's especially true if you're a dev installing and uninstalling all day long.
It has been a game changer for me at least as a first time dev to just run 'nix develop' and have 'cargo build' just work as a declare all the depends, options and such in my flake.nix. You can do the same with 'shell.nix' files so without flakes.
With homebrew, you can have Brewfile that can serve as declarative source of truth.
I try to install all software via homebrew, mise (https://mise.jdx.dev/), and scoop (https://scoop.sh/), and setting up a new machine now takes me minutes. Meanwhile I don't need to deal with Nix language.
> With homebrew, you can have Brewfile that can serve as declarative source of truth.
for homebrew, while Nix configuration is for everything. I never used a Brewfile before, so looked at https://github.com/Lissy93/Brewfile/blob/master/Brewfile but it just looks like a list of packages. What about the configuration for those packages? Or your own custom patches for them? Runtime parameters? Environment variables?
There is so much more going on in the typical developer environment that it doesn't seem like (to me, an uneducated fool) Brewfile would be enough to actually serve as a declarative source of truth, except for Homebrew-specific things.
Versions? Compatibilities? Just pulling latest doesn't cut it.
It has a really good story here imo! Pin nixpkgs to a commit with the version you need, it will pull and use the compatible deps even if you have those deps in different versions being used elsewhere.
I think they were continuing my earlier questions, ie they're asking about those things in Homebrew/Brewfile like I did with my other questions.
Brewfile is definitely the best option when in that ecosystem, but even that falls short by having no explicit versioning. You have to hack in version pinning if you run into an issue with a specific package.
If you like mise, check out devenv for the nix version of it. Its a lot of the nix benefits without a lot of the nix drawbacks (but still some)
> unlock the full disk encryption via SSH
I'd like to read more about this. Are you open to sharing your notes?
This has been my experience as well. Before I switched to NixOS I used ubuntu for 2 years. I never grokked the ways of apt and how or why it would "randomly" brick my system in some way. With NixOS this has never happened. `nix-shell` is dead simple, adding packages to environment is dead simple, never has it bricked my system. The hard part of NixOS is if you want to do advanced things with the actual nix language, and of course the horrible error messages.
In terms of all the linux systems I have used, NixOS seemed to least magical to me in terms of what is happening under the hood.
What?! Insane take. NixOS is where the most "magic" happens, over and under the hood. It brings it's own language!
Simple package based distros like Arch basically just extract archives. Very few packages trigger post-install steps which usually just (re)generate something like initrd.
Afaik, bricking Ubuntu is either due to user error (e.g. mixing incompatible package sources) or the devs released broken/buggy packages...
If you're not making changes to the bootloader it's essentially impossible to brick nixos: updates are fully atomic and every change can be rolled back by booting into an old generation.
This combined with the fact that the full source code for the system is contained within a single monorepo that I can checkout and grep through makes NixOS the easiest to understand and most transparent distro I have ever used.
> updates are fully atomic and every change can be rolled back by booting into an old generation
Well, the updates themselves yeah, but not what data they use. You cannot always rollback database upgrades for example, without also having to rollback the data source of the database. In most cases you're right though.
I'm saying this as someone who is a fan of NixOS and use it on all my servers because I tend to forget what I do if I just ssh in and fix stuff. Although I'm on Arch/CachyOS on all other hardware.
Well to be fair, nix packages (as in the actual packages and not the modules/services) also mostly just extract archives. I mean the entire system is built on the assumption that 'config && make && make install' is how the thing is built.
Then nix layers way more nicety on top by offering modules, but nothing prevents you from ignoring all that and just using installing packages.
> Simple package based distros like Arch basically just extract archives. Very few packages trigger post-install steps which usually just (re)generate something like initrd.
Sure it's simple, just like 80% solving the problem is usually significantly simpler than solving it 100%. Nix (and its generation) is the only package manager that actually works.
Try installing kde and gnome, then uninstalling both and check how many packages remain.
Nix can do that with the whole world with nothing residing.
I'm a relative newbie with Nix, but I recently installed a Gnome extension through Home Manager, and then removed it again. It left some native functionality unusable because the install flipped a pref and the dev had forgotten to revert that on uninstall. They fixed it quickly and it was nice and all, but it's still somewhat unpredictable to me when I will run into such cases.
> Try installing kde and gnome, then uninstalling both and check how many packages remain.
Sounds like a package manager or package problem. Competent package managers (e.g. dnf) remove unneeded packages and all their owned files. Albeit, I think with apt you need to do a manual autoremove to remove orphaned packages.
Not suggesting that they're equivalently powerful compared to nix, but this specific thing shouldn't be a problem with traditional package managers.
A common issue with most package managers is that if you have A installed, and then you install B which depends on C, and that C happens to also be an optional dependency of A, then uninstalling B will not uninstall C as C won't be orphaned (because of A).
That's interesting. I'm surprised. Just some cursory websearching and didn't see anything that gave a solution here for DNF at least. Funky! Seems like there should be a way to deal with this.
Yeah, I have never, ever broken Ubuntu by installing packages. People are talking in this thread like Ubuntu or other distros are super fragile, but they really aren't.
Agree to disagree, I guess.
I've found three "hotspots" that break Ubuntu at a pretty high rate of incidence: 1) Upgrading to the next distro release 2) Installing or updating Nvidia drivers using the built-in management tool 3) Changing to a different kernel "series" (such as switching to a hardware enablement kernel)
In all these cases, I've found that the process stands a solid (hand-waving: 20%) chance of stopping in the middle, requiring manual resolution of broken packages.
I think you uncovered the actual reason without saying it.
- in traditional package manages, the dependency is global, and potentially breaks everything else installed.
- in nixos, package is isolated to the environment and can co-exist on its own, for the purpose of one application.
I never used arch so I can't comment on that.
I'm comparing it to my experience with Ubuntu where packages can have complex install dependency flows that can destroy your system at any time. They also pollute your entire system. This is simple in the same way assembly is a simple way to program. You can do anything and destroy anything. It's essentially spaghetti code but in OS form.
You can say it's "user error" or "packaging error", which is arguably true, but this "user error" and "packaging error" literally does not exist in NixOS. Installing a package is unable to touch anything outside of it's own designed nix store folder. That is why it's so much simpler to understand for me. I can check the nix store path and see what a package has. I can fearlessly install a package without worrying my system will break. I don't have to worry about residual dependencies remaining on my system.
The nix language itself ain't it though, I agree.
They also do a really sloppy job of marking why packages were installed in the initial installation media... A bunch of packages that are not critical get marked as manually installed (so they don't go away if you remove the things that depend on them). At the same time, a bunch of packages that are actually critical don't get marked as manually installed. This means that you can remove a package you don't want and then autoremove will start wanting to uninstall critical parts of your system.
> So do we want to do: We want to build one disk image that boots on x86_64, ARM AArch64, and RISC-V 64-bit. We limit ourselves here to UEFI platforms, which makes this pretty straight forward.
It kind of feels like this should have been the default across all architectures - a single disk that supports multiple instruction sets. 32 bit? 64 bit? Intel x86? Intel x86_64? RISC-V? Imagine how much simpler it would be for the end-user.
I still believe to this day that it has been a massive mistake to not ship ARM devices with some form of BIOS/UEFI chip to allow the system to boot without a bootable image.
That's a thing on ARM. See: ARM SystemReady.
Unfortunately, the adoption has been very slow going because ARM SOC vendors are set in their ways.
This could possibly make a good base for a system recovery USB drive. One 18-headed hammer for all your needs.
I haven't looked deep into it, but my impression is that most system recovery images target just x86_64 and maybe 32-bit x86 if they're cheeky.
Not surprised that it's easy to do with NixOS, but that is a brute force method, it's just easy to implement with the way NixOS works.
I remember I had G4 Mac mini and one of the first gen intel macbook pros - you could have use one of them as a hard-drive and boot from it via Firewire. Work magically.
Is this a valid alternative to Nix Darwin?
https://github.com/nix-darwin/nix-darwin
Great project, but I had to discontinue using it on my dev machine because of some issues getting the correct versions of dependencies installed.
Something about this rubs me the wrong way. You can get completely different behaviour depending on which architecture you boot on.
This is always true for every program written in most languages (very few languages have zero dependencies on the architecture, particularly if you include endianness).
If you want to be really pedantic, you can get completely different behavior if you built your system image in February in C with just:
if (__DATE__[0] == 'F') {
// February
} else {
// Not February
}> I thought this would bring some space savings, because files that are not binary code should be largely the same.
Those ternary blobs tend to be cross-platform, I hear.
The nix store can do hard linking for the hashsum files. The nix store is read-only so you don't get modifications in general - does not really matter for boot disks, but in general.
You have to do extra work to get the hardlink deduplication in the store though:
https://nix.dev/manual/nix/2.20/command-ref/nix-store/optimi...
Unlike in FHS distros where you get some of the separation for free with usr/lib vs usr/share, most nix packages don't have separate store paths for binary vs non-binary files. At most you'll get the headers and build scripts split off in a separate dev path.