General



The name "systemd" is a bit confusing: There is They all come from the systemd project in Red Hat.



Heard somewhere: Systemd is an attempt to organize and structure the "middleware" of Linux, the part above kernel and libc but below DE and applications. This was a fairly unstructured mix of init scripts and daemons and event-handlers and services.

From someone on reddit:
"The industry trend is against scripting and toward declarative systems across the spectrum of management tools. Systemd units are a declarative spec with, notably, event handlers such as restarting a service when it exits."



Wikipedia's "systemd"
systemd diagram
Sahil Suri's "Introducing systemd"
ArchWiki's "systemd"
openSUSE's "The systemd daemon"
GoLinuxCloud's "Beginners Guide on systemd tutorial in Linux"
Carla Schroder's "Understanding and Using Systemd"
David Both's "Learning to love systemd"
neeasade's "systemd user services" (under $HOME/.config/systemd/user)
Paul Brown's "systemd's 'Socket' Units"
Andy's "Starting services only when the network is ready on Debian/systemd"
Trapped inside a tolva's "systemd targets and infrastructure layers"

systemd.slice(5)
David Both's "Managing resources with cgroups in systemd"
systemd.resource-control(5)

Sebastian Jambor's "systemd by example"
systemd.io
freedesktop.org's "systemd System and Service Manager"
Chris Hoffman's "Meet systemd"

Simon Ruderich's "Systemd service hardening"
alegrey91 / systemd-service-hardening
Ctrl blog's "systemd application firewalls by example"

Debian Reference's "Chapter 3. The system initialization"
Jason Wertz's "Basics of the Linux Boot Process" (video) (2013 and pre-systemd, but interesting)
Professor Messer's "Init, Systemd, and Upstart" (video) (2013, not much systemd, but interesting)
Aun Raza's "Systemd Boot Process a Close Look in Linux"
Vivek Gite's "How to switch boot target to text or GUI in systemd Linux"
systemd's "The Discoverable Partitions Specification (DPS)"

David Both's "Manage Linux users' home directories with systemd-homed"

SergeantBiggs' "Credential Management With Systemd"




pidof systemd			# are you using systemd ?

man systemd
man systemd.index
man udev

pstree					# see running processes
systemctl list-units	# show in-memory units, with descriptions
systemctl status		# show all defined units

systemctl status UNITNAME --full --lines 1000
systemctl cat UNITNAME	# see source for a unit

# check syntax of a unit:
systemd-analyze verify /etc/systemd/system/UNIT.service

systemctl status PID	# find out which unit a process belongs to

GUI for systemd:
In KDE, install "kcm_systemd" and then look in System Settings.
In GNOME, maybe use GuillaumeGomez / systemd-manager




Casey Houser's "How to Use Systemd Timers as a Cron Replacement"
blog'o'less's "Using systemd timers instead of cron"
blog'o'less's "More about using systemd timers: reboot"
luqmaan's "Using systemd as a better cron"
Chris Siebenmann article
Richard England's "Systemd Timers for Scheduling Tasks"

ArchWiki's "systemd/Timers: As a cron replacement"
From discussion on stackexchange, why systemd better than cron:
+/-
Checking what your cron job really does can be kind of a mess, but all systemd timer events are carefully logged in systemd journal.

Systemd timers are systemd services with all their capabilities for controlling their resource management, IO, CPU, scheduling, etc.

There can be dependencies on other services.

Services can be started and triggered by different events like user, boot, hardware state changes or for example 5 mins after some hardware plugged in.

Easily enable/disable the whole thing with "systemctl enable/disable" and kill all the job's children with "systemctl start/stop".

Timer events can be scheduled based on finish times of executions some delays can be set between executions.

Communication with other programs is also notable; sometimes it's needed for some other programs to know timers and the state of their tasks.

Disabling a service doesn't mean that other services can't start it any more - masking does that.



Good for development purposes: "systemd-nspawn -x -D /" will create a snapshot of your /, chroot into it, and delete it on exit. Works for XFS and Btrfs.
Pid Eins's "Running a Container off the Host /usr/"



To run an app while denying network access:

systemd-run --scope -p IPAddressDeny=any APPNAME

To run an app using limited amount of memory:

systemd-run --scope -p MemoryMax=500M APPNAME



Using journalctl, from comments on Hacker News:
+/-
Want logs since boot?
journalctl -b
The first boot is -b 0, the previous boot is -b -1.

Boot history ?
journalctl --list-boots
last reboot


Want logs for smartd?
journalctl -u smartd

Want warnings and above?
journalctl -p warning

Want logs since yesterday?
journalctl -S yesterday

Want logs on April 22 between 10 and 11?
journalctl -S "2021-04-22 10:00" -U "2021-04-22 11:00"

Want logs with microsecond timestamps?
journalctl -o short-precise

Or in JSON for parsing?
journalctl -o json

Want tail -f?
journalctl -f

Want only kernel logs?
journalctl -k or journalctl --dmesg

Want to filter the output right away?
journalctl --grep=some_regex

You booted a live environment, but want to look at the installed OS's journal:
journalctl --root=/path/to/mounted_root or journalctl --directory=/path/to/journal

Look at logs from a container you booted using systemd-nspawn (and friends)?
journalctl --machine=container_id

What time intervals do the boot indices (for -b) correspond to?
journalctl --list-boots
Artemist's "Jumping into journald"



Slices, from a Christine Dodrill article:
+/-
Apparently GNOME runs every application in its own systemd slice (which really does explain why GNOME has a hard requirement for systemd as well as why the "Force Quit" button actually seems to work reliably and clean up all the associated clutter in one fell swoop), so you can fetch this stuff from your user journal with "systemctl --user status" to find the slices (type /APPNAME and hit the enter key to search for it) and "journalctl --user -u SLICENAME.slice" to look through their output that way.



To boot into systemd rescue mode, in GRUB edit kernel command line to add "systemd.unit=rescue", then boot. Later to continue booting as normal, run "systemctl isolate graphical.target".





Good



Christine Dodrill's "systemd: The Good Parts"
NotMine's "Archlinux is moving to systemd"
/u/thlst's "Why did ArchLinux embrace Systemd?"
David Edmundson's "Plasma and the systemd startup"
David Both's "Why I support systemd's plan to take over the world"
Pid Eins's "The Biggest Myths"
Jesse Smith's "Deep diving into systemd"

IMO: systemd the service manager does one thing, and does it well: represent/control units of work. It satisfies many uses: managing services at boot/login/logout/shutdown times, letting the user manage services manually, managing services in reaction to events, allowing services/timers to trigger other services. I'd like to see init go away completely, and also have cron be a systemd service (that launchs other services).

From comment on Hacker News:
+/-
It solves a whole lot of problems to me.

  • It gets rid of a system cobbled together from init, inittab (which people rarely remember exists), monit, cron, and inetd/xinetd.

  • It removes a bunch of horrible hacks in init scripts such as calls to sleep.

  • It removes the need for pid files.

  • It removes the need for a lot of stupid boilerplate, like the whole start-stop-daemon stuff in scripts.

  • It allows precisely tracking what belongs to each service. I can easily find out what a random process belongs to.

  • It captures all the log output of each process.

  • It drastically improves the upgrade process -- I never need to look at a 3-way diff of /etc/init.d/apache2 again.

  • As a developer it means I don't need to write a slightly different script for every distro.

  • It makes logging a whole lot pleasant.

  • It allows having user services without having to hack around with cron.

  • It allows a whole bunch of settings to be applied to any random service, without having to edit a shell script and figure out how to stick it in there.

  • It means services run identically when started by hand, without the confusion of a service inheriting my local environment.

  • It makes my VM servers boot faster. And if you're going to pipe up with "I reboot my servers once a year", that doesn't cut it for me. I boot my servers on demand, when they're needed. They adapt to the load.


From comment on Hacker News:
+/-
The perspective from which I care about the advantages is that of a programmer writing daemons: running under a daemon manager (such as, but not limited to, systemd) lets me omit a bunch of code around forking off, daemonisation, automatic restarts, switching uid/gid, logging and rotation. The important thing about these pieces of functionality is that they provide no competitive advantage.

As someone who wants to write a daemon, you need them, but there's no room to delight anyone by doing them well (unlike the actual functionality for which your daemon is being written) and there's plenty of room to mess them up and have people yell at you.

From comments on Lobsters 6/2021:
+/-
> I'm so glad we've moved past trying to do all of this with shell scripts.

This can't be stated strongly enough. I recently had to fix a vendor-provided (not from the particular software in question) init script that spawned a daemon that inherited file descriptors it shouldn't have. There are simply too many ways to just pile on spaghetti that can have strange effects. There were other maintenance tasks that required rewrites of most of these scripts just because critical software changes: going from udevd to eudev.

I cannot stand init scripts.

...

I recently took a look at some software a customer wanted installed on all their Linux servers. It had an undocumented upgrade function that fetched and installed software without any checksums and without verifying HTTPS certificates, the upgrade function was not disabled even when explicitly using the somewhat-documented flag to disallow upgrades. It dropped lines in /etc/rc.d/rc.local, it spawned itself during postinst scripts outside of the init system, and trying to upgrade the package resulted in the software deleting itself. The postinst script also looks for a line in /etc/rc.d/rc.local that suggests that they previously modified /etc/ssh/sshd_config on install ...

From comment on Lobsters 6/2021:
+/-
The big difference [between systemd and others] is that systemd (as well as runit, s6, etc) stay attached to the process, whereas the BSD systems (and OpenRC, traditional Linux init scripts) expect the program to "daemonize".

Aside from whatever problems systemd may or may not have, I feel this model is vastly superior in pretty much every single way. It simplifies almost everything, especially for application authors, but also for the init implementation and system as a whole.

From someone on reddit:
+/-
Systemd From an Arch init maintainer perspective:

I was the primary maintainer for Arch's init scripts for a while and I can share a couple of thoughts.

Arch's init-scripts were incredibly stupid. In their first phase, there was a static set of steps that would be performed on every boot. There was almost no way to adjust the behaviour here. In their second phase, the configured daemons were started in order, which only meant that init scripts were called one after another.

In the early 2000s, that seemed like a good idea and has worked for a while. But with more-complex setups, the shortcomings of that system become apparent.

  • With hardware becoming more dynamic and asynchronous initialization of drivers in the kernel, it was impossible to say when a certain piece of hardware would be available. For a long time, this was solved by first triggering uevents, then waiting for udev to "settle". This often took a very long time and still gave no guarantee that all required hardware was available. Working around this in shell code would be very complex, slow and error-prone: You'd have to retry all kinds of operations in a loop until they succeed. Solution: An system that can perform actions based on events - this is one of the major features of systemd.

  • Init-scripts had no dependency-handling for daemons. In times where only a few services depended on dbus and nothing else, that was easy to handle. Nowadays, we have daemons with far more-complex dependencies, which would make configuration in the old init-scripts-style way hard for every user. Handling dependencies is a complex topic and you don't want to deal with it in shell code. Systemd has it built-in (and with socket-activation, a much better mechanism to deal with dependencies).

  • Complex tasks in shell scripts require launching external helper program A LOT. This makes things very slow. Systemd handles most of those tasks with built-in fast C code, or via the right libraries. It won't call many external programs to perform its tasks.

  • The whole startup process was serialized. Also very slow. Systemd can parallelize it and does so quite well.

  • No indication of whether a certain daemon was already started. Each init script had to implement some sort of PID file handling or similar. Most init scripts didn't. Systemd has a 100% reliable solution for this based on Linux cgroups.

  • Race conditions between daemons started via udev rules, dbus activation and manual configuration. It could happen that a daemon was started multiple times (maybe even simultaneously), which lead to unexpected results (this was a real problem with bluez). Systemd provides a single instance where all daemons are handled. Udev or dbus don't start daemons anymore, they tell systemd that they need a specific daemon and systemd takes care of it.

  • Lack of configurability. It was impossible to change the behaviour of init-scripts in a way that would survive system updates. Systemd provides good mechanisms with machine-specific overrides, drop-ins and unit masking.

  • Burden of maintenance: In addition to the aforementioned design problems, init-scripts also had a large number of bugs. Fixing those bugs was always complicated and took time, which we often did not have. Delegating this task to a larger community (in this case, the systemd community) made things much easier for us.

I realize that many of these problems could be solved with some work, and some were already solved by other SysV-based init systems. There was no system that solved all of these problems and did so in a reliable manner, as systemd does.

So, for me personally, when systemd came along, it solved all the problems I ever had with system initialization. What most systemd critics consider "bloat", I consider necessary complexity to solve a complex problem generically. You can say what you want about Poettering, but he actually realized what the problems with system initialization were and provided a working solution.

I could go on for hours, but this should be a good summary.


From someone on reddit:
+/-
Resolved does two things not available with libc resolvers. First, it has by interface domain routing, so you can have e.g. VPN hosts only in certain domain resolved through its nameserver. And second is that it remembers servers which are down so it doesn't retry them all over again. Plus it has caching, plus dnssec validation.
zbyszek's "systemd-resolved: introduction to split DNS"
From someone on reddit:
+/-
Having two DNS entries put into /etc/resolv.conf only works if you want to have 100% of the DNS configuration done on the DNS server and you trust your DNS server explicitly.

If you want to be able to validate DNS responses, use VPN connections with private IP addresses with DNS names, connect to public access points, or essentially use anything more complicated then a single statically configured Ethernet port for life ... then putting IP addresses in /etc/resolv.conf isn't going to cut it.

You need to have a local resolver service to provide those features in a sane manner.
From someone on reddit:
+/-
Networkd allows you to specify DNS servers for individual network devices, set DNS server priority, and use different DNS servers per subnet or FQDN.





Bad



Dave McKay's "Why Linux's systemd Is Still Divisive After All These Years"




Miscellaneous



Troubleshooting:
"systemctl --failed"
Ritesh Raj Sarraf's "Systemd Service Hang"



V.R.'s "systemd, 10 years later: a historical and technical retrospective"

See "systemd Service" section of my Develop an Application page