What is systemd?

Systemd is the init system that most Linux distros have converged upon as the successor to previous popular init systems SysVinit and upstart. The init system (aka PID 1) is responsible for starting up the rest of the services that the system is going to run and with systemd those services are defined via unit files.

Something I didn’t know about systemd is that it actually has two standard unit file installation locations as well as unit-file specific directories for override snippets and an inheritance model for processing appropriate files in the correct order. I had always assumed that everything just went into /usr/lib/systemd/system/$UNIT_NAME.$TYPE where the package manager dropped in unit files.

Where are unit files defined?

But then I saw a coworker create a unit file at /etc/systemd/system, and saw that systemd was aware of that unit file. So I looked into it and found out that there is the following override priority (using httpd.service as an example here):

/etc/systemd/system/httpd.service.d/override.conf
              ↑
/etc/systemd/system/httpd.service
              ↑
/usr/lib/systemd/system/httpd.service

So what’s going on here and why are there multiple locations for systemd to keep track of and override unit files? Well the default software installation location is the /usr/lib/systemd/system directory. If you modify the default unit file installed by upstream software repositories then the unit file won’t inherit any changes when the software package gets updated. So if you need to override a couple of settings you can generate a unit override snippet with systemctl edit httpd which creates the /etc/systemd/system/httpd.service.d/override.conf override file.

The reasoning behind override snippets makes sense, but makes it more difficult to parse what is actually happening when a service starts. So what if you want to keep the entire unit file in one location but you still want to override the default upstream unit file? That’s where the /etc/systemd/system directory comes in. If for whatever reason you’re starting from scratch then go ahead and drop that homegrown unit file in there.

Or if you want to use the upstream unit file as a baseline run systemctl edit --full httpd to auto-generate a copy of the existing vendor unit file into /etc/systemd/system. Just be aware that you’ll have to use another mechanism to identify any upstream unit file changes and backport them into your own unit file, applying them with systemctl daemon-reload. It is also worth noting that override snippets can also override the contents of full override unit files so you can run into some trouble if you go crazy with overrides.

Commands to help

Pretty much unless you’re packaging software for some downstream customer (e.g. you’re part of youre company’s build and release team) just leave everything in /usr/lib/systemd/system alone and put your override snippets and files in /etc/systemd/system. In summary, here are a few commands that you can use to help manage unit files:

• systemctl show httpd: shows the (IMO poorly) formatted full content of the merged unit file, including defaults that are not explicitly mentioned in the base unit file /usr/lib/systemd/system/httpd.service: default location for installed unit files
• systemctl edit httpd: creates a unit snippet override file at /etc/systemd/system/httpd.service.d/override.conf
• systemctl edit --full httpd: auto-generates a copy of the existing unit file into /etc/systemd/system
• systemctl status httpd: shows which unit file is loaded for the httpd service

And here you can see that the output of systemctl status httpd lets you see exactly what unit files are applied for a specific service:

● httpd.service - The Apache HTTP Server
  Loaded: loaded (/etc/systemd/system/httpd.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/httpd.service.d
          └─override.conf