The name "systemd" is a bit confusing: There is
- systemd the service manager.
- journald, networkd, sd-bus, systemd.slice (I think all mandatory).
- systemd-resolved.service, systemd-boot, systemd-homed.service (all optional).
- consoled, systemd-logind.service, systemd-hostnamed.service, systemd-machined.service, systemd-tmpfiles, systemd-stub, systemd-measure, systemd-cryptenroll, systemd-pcrphase, systemd-creds, many more (not sure if all optional).
Heard somewhere: Systemd is an attempt to organize and structure the "middleware" of Linux, the part above kernel and libc but below DE and applications. This was a fairly unstructured mix of init scripts and daemons and event-handlers and services.
(Relevant: Tony Garnock-Jones' "Breaking Down the System Layer")
From someone on reddit:
"The industry trend is against scripting and toward declarative systems across the spectrum of management tools. Systemd units are a declarative spec with, notably, event handlers such as restarting a service when it exits."
Sahil Suri's "Introducing systemd"
openSUSE's "The systemd daemon"
Debian Wiki's "systemd Services"
GoLinuxCloud's "Beginners Guide on systemd tutorial in Linux"
Carla Schroder's "Understanding and Using Systemd"
SoByte's "Linux Systemd Getting Started"
David Both's "Learning to love systemd"
David Both's "A pragmatic guide to systemd for Linux sysadmins" (eBook)
neeasade's "systemd user services" (under $HOME/.config/systemd/user)
Paul Brown's "systemd's 'Socket' Units"
Andy's "Starting services only when the network is ready on Debian/systemd"
SergeantBiggs' "Path-Based Activation of systemd Units"
Trapped inside a tolva's "systemd targets and infrastructure layers"
milosz's "How to modify systemd service configuration"
Baeldung's "systemd Scope and systemd Slice"
David Both's "Managing resources with cgroups in systemd"
Chris Siebenmann's "Some practical notes on the systemd cgroups/units hierarchies"
Sebastian Jambor's "systemd by example"
freedesktop.org's "systemd System and Service Manager"
Chris Hoffman's "Meet systemd"
Simon Ruderich's "Systemd service hardening"
alegrey91 / systemd-service-hardening
Ctrl blog's "systemd application firewalls by example"
Debian Reference's "Chapter 3. The system initialization"
Jason Wertz's "Basics of the Linux Boot Process" (video) (2013 and pre-systemd, but interesting)
Professor Messer's "Init, Systemd, and Upstart" (video) (2013, not much systemd, but interesting)
Aun Raza's "Systemd Boot Process a Close Look in Linux"
Vivek Gite's "How to switch boot target to text or GUI in systemd Linux"
systemd's "The Discoverable Partitions Specification (DPS)"
David Both's "Manage Linux users' home directories with systemd-homed"
SergeantBiggs' "Credential Management With Systemd"
Baeldung's "Getting Notification When systemd Service Fails"
pidof systemd # are you using systemd ?
systemctl -t help # see types of units
systemctl list-unit-files # see all units
pstree # see running processes
systemctl list-units # show in-memory units, with descriptions
systemctl status # show all defined units
systemctl status UNITNAME --full --lines 1000
systemctl cat UNITNAME # see source for a unit
systemctl edit UNITNAME # create an override/addition for a unit
# check syntax of a unit:
systemd-analyze verify /etc/systemd/system/UNIT.service
# doesn't actually list all targets:
systemctl list-units --type target
# see for example:
systemctl cat rescue.target
systemctl status PID # find out which unit a process belongs to
systemd-cgtop # see resource use of services
GUI for systemd:
In KDE, install "kcm_systemd" and then look in System Settings.
In GNOME, maybe use GuillaumeGomez / systemd-manager
Casey Houser's "How to Use Systemd Timers as a Cron Replacement"
blog'o'less's "Using systemd timers instead of cron"
blog'o'less's "More about using systemd timers: reboot"
luqmaan's "Using systemd as a better cron"
Chris Siebenmann article
Richard England's "Systemd Timers for Scheduling Tasks"
ArchWiki's "systemd/Timers: As a cron replacement"
From discussion on stackexchange, why systemd better than cron:
Checking what your cron job really does can be kind of a mess, but all systemd timer events are carefully logged in systemd journal.
Systemd timers are systemd services with all their capabilities for controlling their resource management, IO, CPU, scheduling, etc.
There can be dependencies on other services.
Services can be started and triggered by different events like user, boot, hardware state changes or for example 5 mins after some hardware plugged in.
Easily enable/disable the whole thing with "systemctl enable/disable" and kill all the job's children with "systemctl start/stop".
Timer events can be scheduled based on finish times of executions some delays can be set between executions.
Communication with other programs is also notable; sometimes it's needed for some other programs to know timers and the state of their tasks.
Disabling a service doesn't mean that other services can't start it any more - masking does that.
Good for development purposes: "systemd-nspawn -x -D /" will create a snapshot of your /, chroot into it, and delete it on exit. Works for XFS and Btrfs.
Pid Eins's "Running a Container off the Host /usr/"
To run an app while denying network access:
systemd-run --scope -p IPAddressDeny=any APPNAME
To run an app using limited amount of memory:
systemd-run --scope -p MemoryMax=500M APPNAME
To run an app and prevent any part of it from being swapped out:
"Run process in a cgroup with memory.swap.max set to 0. systemd exposes this through the MemorySwapMax= directive. Available only if you are using the unified cgroup hierarchy. [May not cover shared pages if owned by another cgroup. mlock() is the true way for an app to prevent pages from being swapped.]"
Using journalctl, mostly from comments on Hacker News:
Want logs since boot?
sudo journalctl -b
The first boot is -b 0, the previous boot is -b -1.
Want end of log for last boot ?
sudo journalctl -b -1 -e
Boot history ?
sudo journalctl --list-boots
Want logs for smartd?
sudo journalctl -u smartd
Want warnings and above?
sudo journalctl -p warning
Want logs since yesterday?
sudo journalctl -S yesterday
Want logs on April 22 between 10 and 11?
sudo journalctl -S "2021-04-22 10:00" -U "2021-04-22 11:00"
Want logs with microsecond timestamps?
sudo journalctl -o short-precise
Or in JSON for parsing?
sudo journalctl -o json
Want tail -f?
sudo journalctl -f
Want only kernel logs?
sudo journalctl -k or journalctl --dmesg
Want to filter the output right away?
sudo journalctl --grep=some_regex
You booted a live environment, but want to look at the installed OS's journal:
sudo journalctl --root=/path/to/mounted_root or journalctl --directory=/path/to/journal
Look at logs from a container you booted using systemd-nspawn (and friends)?
sudo journalctl --machine=container_id
What time intervals do the boot indices (for -b) correspond to?
sudo journalctl --list-boots
Add a message to the journal?
echo Hello There | systemd-cat
Slices, from a Christine Dodrill article:
Apparently GNOME runs every application in its own systemd slice (which really does explain why GNOME has a hard requirement for systemd as well as why the "Force Quit" button actually seems to work reliably and clean up all the associated clutter in one fell swoop), so you can fetch this stuff from your user journal with "systemctl --user status" to find the slices (type /APPNAME and hit the enter key to search for it) and "journalctl --user -u SLICENAME.slice" to look through their output that way.
To boot into systemd rescue mode, in GRUB edit kernel command line to add "systemd.unit=rescue" (or use Kernelstub to do it), then boot. Later to continue booting as normal, run "systemctl isolate graphical.target".
- Does things that init scripts can't do: react to events, do operations in parallel
(but init scripts can do this with startpar ?), let
user start/stop daemons, set limits to the resource usage of a service, have one
service depend on result of another (but init scripts can do this with insserv ?).
- Pushes control away from the init-project and towards the distro maintainers.
It's easier to enable/disable/add/delete services than to rewrite the init scripts.
- User is less likely to bork the system by making a mistake while
adding a new service than while editing init scripts ? More modular.
- Has "targets", which are finer-grained and more flexible than init's "run-levels".
- Various parts solve large-scale problems for enterprises. Desktop users may
not understand this.
NotMine's "Archlinux is moving to systemd"
/u/thlst's "Why did ArchLinux embrace Systemd?"
David Edmundson's "Plasma and the systemd startup"
David Both's "Why I support systemd's plan to take over the world"
Pid Eins's "The Biggest Myths"
Jesse Smith's "Deep diving into systemd"
IMO: systemd the service manager does one thing, and does it well: represent/control units of work. It satisfies many uses: managing services at boot/login/logout/shutdown times, letting the user manage services manually, managing services in reaction to events, allowing services/timers to trigger other services. I'd like to see init go away completely, and also have cron be a systemd service (that launches other services).
From comment on Hacker News:
It solves a whole lot of problems to me.
- It gets rid of a system cobbled together from init, inittab (which people rarely remember exists),
monit, cron, and inetd/xinetd.
- It removes a bunch of horrible hacks in init scripts such as calls to sleep.
- It removes the need for pid files.
- It removes the need for a lot of stupid boilerplate, like the whole start-stop-daemon stuff in scripts.
- It allows precisely tracking what belongs to each service. I can easily find out
what a random process belongs to.
- It captures all the log output of each process.
- It drastically improves the upgrade process -- I never need to look at a 3-way diff
of /etc/init.d/apache2 again.
- As a developer it means I don't need to write a slightly different script for every distro.
- It makes logging a whole lot pleasant.
- It allows having user services without having to hack around with cron.
- It allows a whole bunch of settings to be applied to any random service, without
having to edit a shell script and figure out how to stick it in there.
- It means services run identically when started by hand, without the confusion of a
service inheriting my local environment.
- It makes my VM servers boot faster. And if you're going to pipe up with
"I reboot my servers once a year", that doesn't cut it for me. I boot my servers on demand,
when they're needed. They adapt to the load.
From comment on Hacker News:
The perspective from which I care about the advantages is that of a programmer writing daemons: running under a daemon manager (such as, but not limited to, systemd) lets me omit a bunch of code around forking off, daemonisation, automatic restarts, switching uid/gid, logging and rotation. The important thing about these pieces of functionality is that they provide no competitive advantage.
As someone who wants to write a daemon, you need them, but there's no room to delight anyone by doing them well (unlike the actual functionality for which your daemon is being written) and there's plenty of room to mess them up and have people yell at you.
From comments on Lobsters 6/2021:
> I'm so glad we've moved past trying to do all of this with shell scripts.
This can't be stated strongly enough. I recently had to fix a vendor-provided (not from the particular software in question) init script that spawned a daemon that inherited file descriptors it shouldn't have. There are simply too many ways to just pile on spaghetti that can have strange effects. There were other maintenance tasks that required rewrites of most of these scripts just because critical software changes: going from udevd to eudev.
I cannot stand init scripts.
I recently took a look at some software a customer wanted installed on all their Linux servers. It had an undocumented upgrade function that fetched and installed software without any checksums and without verifying HTTPS certificates, the upgrade function was not disabled even when explicitly using the somewhat-documented flag to disallow upgrades. It dropped lines in /etc/rc.d/rc.local, it spawned itself during postinst scripts outside of the init system, and trying to upgrade the package resulted in the software deleting itself. The postinst script also looks for a line in /etc/rc.d/rc.local that suggests that they previously modified /etc/ssh/sshd_config on install ...
From comment on Lobsters 6/2021:
The big difference [between systemd and others] is that systemd (as well as runit, s6, etc) stay attached to the process, whereas the BSD systems (and OpenRC, traditional Linux init scripts) expect the program to "daemonize".
Aside from whatever problems systemd may or may not have, I feel this model is vastly superior in pretty much every single way. It simplifies almost everything, especially for application authors, but also for the init implementation and system as a whole.
From someone on reddit:
Systemd From an Arch init maintainer perspective:
I was the primary maintainer for Arch's init scripts for a while and I can share a couple of thoughts.
Arch's init-scripts were incredibly stupid. In their first phase, there was a static set of steps that would be performed on every boot. There was almost no way to adjust the behaviour here. In their second phase, the configured daemons were started in order, which only meant that init scripts were called one after another.
In the early 2000s, that seemed like a good idea and has worked for a while. But with more-complex setups, the shortcomings of that system become apparent.
- With hardware becoming more dynamic and asynchronous initialization of drivers in
the kernel, it was impossible to say when a certain piece of hardware would be available.
For a long time, this was solved by first triggering uevents, then waiting for
udev to "settle". This often took a very long time and still gave no guarantee
that all required hardware was available. Working around this in shell code would
be very complex, slow and error-prone: You'd have to retry all kinds of operations
in a loop until they succeed. Solution: An system that can perform actions based on
events - this is one of the major features of systemd.
- Init-scripts had no dependency-handling for daemons. In times where only a few
services depended on dbus and nothing else, that was easy to handle. Nowadays,
we have daemons with far more-complex dependencies, which would make configuration
in the old init-scripts-style way hard for every user. Handling dependencies
is a complex topic and you don't want to deal with it in shell code. Systemd has
it built-in (and with socket-activation, a much better mechanism to deal with
- Complex tasks in shell scripts require launching external helper program A LOT.
This makes things very slow. Systemd handles most of those tasks with built-in
fast C code, or via the right libraries. It won't call many external programs
to perform its tasks.
- The whole startup process was serialized. Also very slow. Systemd can parallelize
it and does so quite well.
- No indication of whether a certain daemon was already started. Each init script
had to implement some sort of PID file handling or similar. Most init scripts didn't.
Systemd has a 100% reliable solution for this based on Linux cgroups.
- Race conditions between daemons started via udev rules, dbus activation and
manual configuration. It could happen that a daemon was started multiple times
(maybe even simultaneously), which lead to unexpected results (this was a real
problem with bluez). Systemd provides a single instance where all daemons are
handled. Udev or dbus don't start daemons anymore, they tell systemd that they
need a specific daemon and systemd takes care of it.
- Lack of configurability. It was impossible to change the behaviour of init-scripts
in a way that would survive system updates. Systemd provides good mechanisms
with machine-specific overrides, drop-ins and unit masking.
- Burden of maintenance: In addition to the aforementioned design problems,
init-scripts also had a large number of bugs. Fixing those bugs was always
complicated and took time, which we often did not have. Delegating this task
to a larger community (in this case, the systemd community) made things much
easier for us.
So, for me personally, when systemd came along, it solved all the problems I ever had with system initialization. What most systemd critics consider "bloat", I consider necessary complexity to solve a complex problem generically. You can say what you want about Poettering, but he actually realized what the problems with system initialization were and provided a working solution.
I could go on for hours, but this should be a good summary.
From someone on reddit:
Resolved does two things not available with libc resolvers. First, it has by interface domain routing, so you can have e.g. VPN hosts only in certain domain resolved through its nameserver. And second is that it remembers servers which are down so it doesn't retry them all over again. Plus it has caching, plus dnssec validation.
From someone on reddit:
Having two DNS entries put into /etc/resolv.conf only works if you want to have 100% of the DNS configuration done on the DNS server and you trust your DNS server explicitly.
If you want to be able to validate DNS responses, use VPN connections with private IP addresses with DNS names, connect to public access points, or essentially use anything more complicated then a single statically configured Ethernet port for life ... then putting IP addresses in /etc/resolv.conf isn't going to cut it.
You need to have a local resolver service to provide those features in a sane manner.
Networkd allows you to specify DNS servers for individual network devices, set DNS server priority, and use different DNS servers per subnet or FQDN.
- Added a second system log (the journal) to the system, and it's binary, not text.
[But for reasons: "Introducing the Journal"]
- Has its own DNS resolver.
[But maybe it's a good thing: zbyszek's "systemd-resolved: introduction to split DNS"]
From someone on reddit:> Why does systemd have DNS in it ?
Because in certain situations they need to have it very early in the boot process, before other stuff is up and running. In most cases it isn't needed and thus isn't used, but when it is needed it is very useful.
That is the big thing a lot of people don't seem to get: systemd provides a lot of optional components that serve niche use-cases but are turned off by default. In most cases those aren't needed and aren't used, but they are there when someone really needs them.
> Something being turned off by default doesn't
> make it any easier to understand the code,
> or understand which DNS specification takes
> precedence over which other one.
- Runs all services in one process ? Or dispatches all from one process ? Not sure.
"ps axjf | grep `pidof systemd`"
- Now is trying to control the definition of the home directory ?
[But there's a good case to be made for it: article1, article2]
- Also has time-sync ?
- Has a systemd.offline-updates that is very reminiscent of Windows updating,
in that it forces two OS restarts to apply a set of updates.
- An overly-heavy solution for situations where a very simple init system would suffice
(such as a container that contains multiple processes).
- Is Linux-only, so I guess not available on BSD and others.
- Is glibc-specific, so not supported on systems that use musl or other C standard libs ?
So not on Alpine, Void, Android ?
unixsheikh's "systemd isn't safe to run anywhere"
Dave McKay's "Why Linux's systemd Is Still Divisive After All These Years"
Cat Fox Life's "systemd through the eyes of a musl distribution maintainer"
Ritesh Raj Sarraf's "Systemd Service Hang"
If system boot is slow:
"sudo systemd-analyze blame"
"sudo systemd-analyze critical-chain"
[But see Thomas Stringer's "Understanding systemd Critical Chains"]
"journalctl -xp 0..3 -b"
"systemctl --user show-environment"
From someone on reddit:
"This environment is shared across sessions, so an environment saved in one session is used as the starting point for another session."
V.R.'s "systemd, 10 years later: a historical and technical retrospective"
See "systemd Service" section of my Develop an Application page