SystemD Service Hardening
roguesecurity.dev291 points by todsacerdoti 3 days ago
291 points by todsacerdoti 3 days ago
Automatic systemd service hardening guided by strace profiling
A nice thing I found is that if you do (which I see they did not in the examples)
# ProtectSystem=
you can do TemporaryFileSystem=/:ro
BindReadOnly=/usr/bin/binary /lib /lib64 /usr/lib usr/lib64 <paths you want to read>
And essentially just including the binary and the path you want available. ProtectSystem= is currently not compatible with this behavior.EDIT: More info here: https://github.com/systemd/systemd/issues/33688
Seems that might be an issue for something that wants to e.g. send an e-mail when an error occurs?
Much better article with very real tips about what options to try than yesterday's (weirdly flagged/dead?) post on the topic. Which while I really enjoyed lacked substance; I was in the comments trying to provide a more useful basis with some real examples, but this is an exemplary list of awesome ways systemd can easily quickly readily provide aassive boost to isolation & security. Great write up!
Yesterday's, just in case: https://us.jlcarveth.dev/post/hardening-systemd.md https://news.ycombinator.com/item?id=44928504
Maybe fix the certificate issue on the site. Some browser doesnt event let one go forward with a bad cert.
Nitpick and title correction: The proper spelling of systemd is systemd, not SystemD. According to their brand page:
Yes, it is written systemd, not system D or System D, or even SystemD. And it isn't system d either. Why? Because it's a system daemon, and under Unix/Linux those are in lower case, and get suffixed with a lower case d. And since systemd manages the system, it's called systemd.
Interesting! I’ve mostly seen it written as systemD. I wonder why that seems to be so popular…
Many of the more juvenile systemd haters used to deliberately spell it that way.
As an insult, it was rather less successful than the "Micro$oft" / "Slowaris" / "HP-SUX" mockery from the 1990s - but it did manage to sow enough confusion that that it still pops up regularly today, even in contexts that are otherwise neutral or positive about it.
That’s funny, I’m not even sure what the D is supposed to expand to, in that insult. What a silly and lame thing to use as an insult.
I’ve been using it because having some random letter capitalized seems to be the totally unsurprising for this sort of plumbing software. (And by plumbing, I mean: very useful and helpful boring stuff that deals with messy problems that I’m happy not to care about, just to be clear that I mean it positively, haha).
Thanks for sharing this. It looks like you can also use "systemd-analyze" with the "--user" flag to inspect systemd user units as well ("systemd-analyze --user security"). I've started using systemd more now that I've transitioned my containers to Podman, and this will be a helpful utility to improve the security of my systemd unit/container services.
And that's something that's impossible to do with old init scripts, that are all unique in their way and not uniform at all.
You can ofcourse achieve all these things in your init scripts which are unique in their way and not uniform at all, just to give credit where credit is due. But systemd makes it practical to use our beloved kernel and it's features in an uniform and standard way... :)
I started my Linux journey so late I can't imagine living without systemd, the few systems I've encountered without systemd are such a major PITA to use.
I recently discovered "unshare" which I could use to remount entire /nix RW for some hardlinking shenanigans without affecting other processes.
systemd is so good, warty UX when interacting with it but the alternative is honestly Windows in my case.
Aside, I'm quite surprise that systemd won out, given all the hate for so many years. You got nearly every forum and HN thread that even tangently mention systemd, flooded with people proclaiming systemd to be the spawn of the devil "forced down their throat". At the same time I noticed that a lot of regular Linux users (like developers that just want to deploy stuff) seemed to like systemd unit files. I guess all those complainers, however vocal, were neither the majority, nor the people who actually maintained distributions.
This feeling is particular striking for me, because I once worked on a Linux project with the aim of improving packaging and software distribution. We also got a lot of hate, mainly for not being .deb or.rpm, and it looked to me as if the hate was a large reason for the failure of the project.
Systemd won because for all the deserved hate it solves real problems that nobody else was really trying (or their attempts were worse). There are some really good points to systemd - you just have to takes the bad with it.
My feeling, without evidence, is that a lot of the hate is vocal people who don't like this thing that does stuff differently to what they're used to. I get the feeling, I still understand /etc/init.d and runlevel symlinks and /var/log better than systemd, but that's because I have many more years of experience interacting with it, breaking it, fixing it. Whenever I have to do stuff with systemd it's a bit annoying to have to go learn new stuff, but when I do I find it reasonably straightforward. Just different, and most likely better.
A lot of the hate was because the main developers interactions wiht just about everyone was horrible, the code was opaque at best most of the time and constantly changed, scope creep, and the feeling that it was a massive power grab by Red Hat.
It can be argued that it didn't solve very many problems and added a huge amount of complexity.
The problem with the complaints is that they're all made up. Well, except maybe developer interactions - assholes can be assholes.
systemd isn't opaque, it's open-source. systemd is objectively less opaque than init scripts, because it's very well documented. Init scripts are not.
Sure, you can read them. But then you'd realize that glued together init scripts just re-implement systemd but buggier and slower, at which point you might as well just read the systemd source. Or, better yet, the documentation.
systemd ALSO does not constantly change. The init system has been virtually untouched in a decade, save for bug fixes and a few new features. Your unit files will "just work", across many years and many distros. Yes, systemd is more portable than init scripts.
systemd ALSO does not have any scope creep. Here, people get confused between systemd-init, and systemd the project.
systemd-init is just an init system. Nothing more, nothing less, for a long time now, and forever. There is no scope creep, the unix principle is safe, yadda yadda yadda.
systemd coincidentally is also the name of a project which includes many binaries. All of those binaries are optional. They aren't statically linked, they're not even dynamically linked - they communicate over IPC like every other program on your computer.
systemd is also not complex, it's remarkably simple. systemd unit files are flat, declarative config files. Bash is a turing-complete scripting language with more footguns than C++. Which sounds more complex?
> systemd[-init] ALSO does not constantly change. The init system has been virtually untouched in a decade...
Sure, I'll bite. It'll be more interesting than watching some stupid Twitch streamer.
Gentoo Linux was using OpenRC back in 2002. Looking at the copyright notice in relevant source files, it looks like OpenRC is a project that has been under development since 1999, so I'd expect it was in use back in 1999. However, I will use 2002 as the start date for this discussion because that's when I started using it.
The simple OpenRC service file I mention in footnote 3 in [0] is syntactically identical to the syslogd service file added in to the OpenRC repo back in this commit in late 2007 [1]. The commit that appears to add support for 'command_args' and friends is earlier that day.
So, four years before SystemD's experimental release, the minimal OpenRC service file (that I talk about in [0]) was no more complicated than what would become the minimal SystemD service file no less than four years later. What's more, the more-verbose syntax for service files written in 2002 was supported by 2007's OpenRC, and continues to be supported by 2025's OpenRC.
23 years is quite a bit longer than 15.
> systemd is also not complex, it's remarkably simple. systemd unit files are flat, declarative config files.
See above (and below).
> Here, people get confused between systemd-init, and systemd the project.
In that case, it doesn't do you credit to use "systemd-init" and "systemd" interchangably in your commentary. SystemD absolutely has scope creep. systemd-init... well, I think I remember when it wasn't possible to have it rexecute itself for a no-reboot upgrade of PID 1. And does it still have a dependency on DBus, or did they see sense and get rid of that?
[0] <https://news.ycombinator.com/item?id=44945789>
[1] <https://github.com/OpenRC/openrc/commit/3ec2cc50261f37b76e0e...>
systemd, the project, isn't a binary. Its dozens of binaries. It can't have scope creep because that's impossible - they're literally dozens of separate things.
We call that the unix principle, lol.
Saying systemd has scope creep is like saying GNU has scope creep because they have a compiler and a text editor. Makes no fucking sense.
I also don't consider a dependency on dbus "scope creep". It has to communicate over IPC - okay, don't reinvent the wheel, just use dbus. Every program ever supports dbus if it has a public API over IPC. Sorry if that bothers you.
And sure, maybe OpenRC is just as simple as systemd, but the reality is every distro chose systemd and that's that, and for MOST of them they switched from primarily scripts to unit files.
That is a HUGE reduction in complexity. HUGE.
Scope creep as in SystemD is now: the init daemon (systemd-init & systemctl), device manager (systemd-udevd), replaced cron, mount point manager, hostname and time/date controller (hostnamectl, timedatectl), container management (systemd-nspawn, machinectl), session manager (systemd-logind), and of course... the horrific syslog replacement (journald).
It's made Linux more Mac/Windows-like. When it works, it works very well... but when it breaks... good luck figuring out anything.
I guess that's OK for a "desktop" but for a server it's a huge pain in the butt.
> systemd, the project, isn't a binary. Its dozens of binaries. It can't have scope creep because that's impossible - they're literally dozens of separate things.
I suppose you don't understand (or are pretending not to understand) what "scope creep" means. Oh well.
> I also don't consider a dependency on dbus "scope creep".
I also don't consider a dependency on DBus scope creep. I consider making PID 1 crash whenever DBus needs to restart because of a upgrade/security fix/etc. in DBus or one of the libraries it links against to be a fantastically poor design decision.
> And sure, maybe OpenRC is just as simple as systemd...
It's far simpler. It concerns itself with bringing up and supervising services. It doesn't contain a DNS resolver, a (subtly buggy) Cron, a syslog daemon, and the many other things SystemD has decided to (whether correctly or incorrectly) reimplement.
yes, especially the scope and power creep, which is antithetical to what unix was all about. Which was doing one thing, and doing it well. What started as a neat way to start servers in parallel, as systemd handles the sockets, now can control your home directory. Like what?
One of my earliest memories of using systemd involved logs being corrupted. journalctl would just say the file is corrupted but wouldn't even print the name of the file -- I had to resort to strace. That left a real bad taste in my mouth.
My systemd grumpiness was mostly due to only just (nearly) finishing an upstart migration. The thought of another one so soon after wasn't fun, even if I liked some of the new features. Those days are over though, and I'm glad there's a mostly unified approach.
> I guess all those complainers, however vocal, were neither the majority, nor the people who actually maintained distributions.
This matched my experience: there were a few vocal haters who were very loud but tended not to be professional sysadmins or shipping binaries to other people, and they didn’t have a realistic alternative. If you distributed or managed software, you had a single, robust solution for keeping daemons running with service accounts, restarts, dependencies, etc. for Windows NT circa 1993 and macOS in 2005 so Linux not having something comparable was just this ongoing source of paper cuts which caused some Linux shops to have unexpected, highly visible downtime (e.g. multiple times I saw data center outages where all of the Windows stuff and the properly configured Upstart/SystemD stuff come up after retrying but high-profile apps using SysV init stayed down for hours because the admins had to clean it up by hand).
Anyone who packaged software was also happy to stop supporting different combinations of buggy shell scripts and utilities, too – every RPM I built went from hundreds of lines of .sh to a couple dozen lines of better systemd. systemd certainly isn’t perfect but if you had an actual job to do you were going to look at systemd as the best path to reduce that overhead.
Having the backing of RedHat certainly didn't hurt... I don't care either way, although I still think OpenRC style scripts are much easier to debug and sometimes have more elegant solutions (templates vs symbolic links).
I can't say I know what OpenRC really does, but at it's core systemd is just a glorified script runner as well? You have ExecCondition, ExecStartPre, ExecStart, ExecStartPost, ExecReload, ExecStop, ExecStopPost which are all quite self explanatory, this alongside a billion optional parameters which I wouldn't know how to implement in scripts without endless hours of dread.
I remember an old Debian machine with /etc/init.d/something [start|stop|reload|restart] but I can't recall being able to automatically restart services or monitor status easily. (I didn't speak $shell well back then either)
systemd tries to avoid scripts as much as possible.
/etc/init.d/whatever were all shell scripts, and they all had to implement all the features themselves. So `/etc/init.d/foo restart` wasn't guaranteed to work for every script, because "restart" is something that each script handled individually. And maybe this one just didn't bother implementing it.
There's no good status monitoring in sysV because it's all convenience wrappers, not a coherent system.
I've been reading through how many NixOS modules use systemd units and there's a lot of scripts being executed, the final line execs the service (if there is one, NixOS uses systemd for maintenance tasks, refreshing certificates and many more things). While NixOS doesn't speak for the broader community what I'm trying to say is that it can execute anything, if that's a script or a daemon doesn't matter as long as it works for you.
Thanks for the sysV explanation, it sounds worse to me.
> /etc/init.d/whatever were all shell scripts, and they all had to implement all the features themselves.
A minimal SystemD service file and a minimal OpenRC service file are equally complex.
Here's the OpenRC service file for ntpd (which is not a minimal service file, but is pretty close):
#!/sbin/openrc-run
description="ntpd - the network time protocol daemon"
pidfile="/var/run/ntpd.pid"
command="/usr/sbin/ntpd"
command_args="${NTPD_OPTS}"
command_args_background="-p ${pidfile}"
command_args_foreground="-n"
depend() {
use net dns logger
after ntp-client
}
start_pre() {
if [ ! -f /etc/ntp.conf ] ; then
eerror "Please create /etc/ntp.conf"
return 1
fi
return 0
}
'depend' handles service dependency declaration and start/stop ordering (obviously).'start_pre' is a sanity check that could be removed, or reduced to calling an external script (just like -IIRC- systemd forces you to do). There are _pre and _post hooks for both start and stop.
For a service that has no dependencies on other services, backgrounds itself, and creates a pidfile automatically, the smallest OpenRC service file is four non-blank lines: the '#!/sbin/openrc-run' shebang followed by lines declaring 'pidfile', 'command', and 'command_args'. A program that runs only in the foreground adds one more line, which tells OpenRC to handle daemonizing the thing and writing its pidfile: 'command_background="true"'. See [3] for an example of one such service file.
If you want service supervision, it's as simple as adding 'supervisor=supervise-daemon', and ensuring that your program starts in the foreground. If it doesn't foreground itself automatically, then adding 'command_args_foreground=<Program Foregrounding Args>' will do the trick.
If you're interested in more information about OpenRC service file syntax, check out the guide for them at [0], and for a lot more information, the manual for openrc-run at [1]. For supervision, check out the supervision guide at [2].
[0] <https://github.com/OpenRC/openrc/blob/master/service-script-...>
[1] <https://man.uex.se/8/openrc-run>
[2] <https://github.com/OpenRC/openrc/blob/master/supervise-daemo...>
[3] The OpenRC service file for the 'cups-browsed' service (which is a program that does not daemonize itself) is this:
#!/sbin/openrc-run
pidfile="/run/cups-browsed.pid"
command="/usr/sbin/cups-browsed"
command_background="true"
depend() {
need cupsd avahi-daemon
}
So on one hand, yes, that's a vast improvement over SysV.
On the other hand, no sir, I still don't like it. It looks very much like Bash. I'm not very fond of Bash to start with and it might not even be actual Bash? Can't tell from the manpage.
But scrolling down to the bottom of the manpage I see a pretty long sample script, and that's exactly what I want to see completely gone. I don't want to look at a 3-way way merge of a service during an upgrade ever again and try and figure out what's all that jank doing. IMO if any of that shell scripting has any reason to be in a service file, it's a bug to be fixed.
My ideal is the simple systemd services: description, dependencies, one command to start, done. No jank with cleaning up temp files, or signals, or pid files (can they please die already), or any of that.
And one of the nice things about systemd services not being a script is that overrides are straightforward and there's never any diffs involved.
> My ideal is the simple systemd services: description, dependencies, one command to start, done.
> ...
> But scrolling down to the bottom of the manpage I see a pretty long sample script, and that's exactly what I want to see completely gone.
First: I know that you read my comment. The two OpenRC service files that I embedded in it are typical service files. The one in the man page serves as a feature demo, as is appropriate for a man page. [0]
Second: necessary complexity has to live somewhere. Complex OpenRC service files are complex nearly always because the work that their author is doing needs to get done. As folks observe, SystemD service files often are used to invoke a bunch of scripts. As I have demonstrated, you can choose to write OpenRC service files in exactly the same style as you'd use to write a SystemD service file. [1] But if you choose to write OpenRC service files in only that style, and if you need to do complex things as part of service lifecycle management, then you'll need to do what SystemD forces you to do and push that logic out to separate scripts/programs.
Necessary complexity has to live somewhere.
> I don't want to look at a 3-way way merge of a service during an upgrade ever again and try and figure out what's all that jank doing.
I've been using Gentoo Linux for twenty-three years. In the handful of times I've had to examine a service file update, the changes to it have never not been straightforward.
> ...it might not even be actual Bash? Can't tell from the manpage.
It's sh, handled by openrc-run. I'll quote from the service-script-guide that I linked to previously. See [2]:
Service scripts are shell scripts. OpenRC aims at using only the standardized
POSIX sh subset for portability reasons. The default interpreter (build-time
toggle) is /bin/sh, so using for example mksh is not a problem.
OpenRC has been tested with busybox's sh, ash, dash, bash, mksh, zsh and
possibly others. Using busybox's sh has been difficult as it replaces
commands with builtins that don't offer the expected features.
The interpreter for service scripts is #!/sbin/openrc-run. Not using this
interpreter will break the use of dependencies and is not supported.
[0] You noticed that the commands added to the service file are named "eat" and "drink"? If nothing else indicated to you that this was a feature demo, that should have.[1] It's a lot less work to write them in this "key, value" style, and it's obviously good to do so whenever reasonably possible.
[2] <https://github.com/OpenRC/openrc/blob/master/service-script-...>