Manage daemons as user with systemd

Starting with Ubuntu Xenial, systemd is now used as the init system. In this article, we explain how one can interact with systemd and manage your user space services.

In addition to the system wide instance, systemd provides user specific instances which allow users to run services or applications as themselves. Systemd supports different types of units, with the service unit being the most common one, representing a service or daemon. Systemd comes with a variety of new approaches, that especially target the observability and the monitoring of the started process. Log outputs can be sent to STDOUT/STDERR, which will be picked up by systemd and be forwarded to the "journal".

User Systemd Status

The command systemctl --user status shows all processes that are managed by systemd for the current user. The following example shows the systemd user instance (init.scope) and a memcached server (mymemcached.service).

www-data@nine01-test:~ $ systemctl --user status
● nine01-test
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Fri 2021-09-17 17:12:02 CEST; 4 months 10 days ago
   CGroup: /user.slice/user-33.slice/user@33.service
           ├─init.scope
           │ ├─1634 /lib/systemd/systemd --user
           │ └─1647 (sd-pam)
           └─mymemcached.service
             └─106393 /usr/bin/memcached -A -m 32 -p 11212 -u www-data -l 127.0.0.1 -P /home/www-data/memcached.pid

If you want to use the command systemctl in your crontab or within scripts, you'll need to set the environment variable XDG_RUNTIME_DIR:

export XDG_RUNTIME_DIR=/run/user/$UID

Managing a Service

You can use the following commands to check the status, to start, stop and restart a service.

Status

You can get more information about the runtime status of a particular service by appending the service name to the status command. The status of the unit, a list of processes and the last lines of the journal are displayed:

www-data@nine01-test:~ $ systemctl --user status mymemcached.service
● mymemcached.service - Custom memcached on port 11212
   Loaded: loaded (/home/www-data/.config/systemd/user/mymemcached.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-01-26 13:27:43 CET; 24h ago
     Docs: /a/Ws7sDLPoiTo
 Main PID: 106393 (memcached)
   CGroup: /user.slice/user-33.slice/user@33.service/mymemcached.service
           └─106393 /usr/bin/memcached -A -m 32 -p 11212 -u www-data -l 127.0.0.1 -P /home/www-data/memcached.pid

Jan 26 13:27:43 nine01-test systemd[1634]: Started Custom memcached on port 11212.

Start

To start a service, use the start command:

www-data@nine01-test:~ $ systemctl --user start mymemcached.service

Stop

To stop a service, use the stop command:

www-data@nine01-test:~ $ systemctl --user stop mymemcached.service

Restart

To restart a service, use the restart command:

www-data@nine01-test:~ $ systemctl --user restart mymemcached.service

Display the Journal / Logs

journalctl is used to display logs from the journal. The journal can be filtered for a specific services using the parameter --user-unit:

www-data@nine01-test:~ $ journalctl --user-unit=mymemcached.service

Jan 26 13:27:43 nine01-test systemd[1634]: Started Custom memcached on port 11212.

Enable and Disable Services

To start a service at boot, use the enable command:

www-data@nine01-test:~ $ systemctl --user enable mymemcached.service
Created symlink from /home/www-data/.config/systemd/user/default.target.wants/mymemcached.service to /home/www-data/.config/systemd/user/mymemcached.service.

The command systemctl --user is-enabled can be used to check if a service has already been activated.

www-data@nine01-test:~ $ systemctl --user is-enabled mymemcached.service
enabled

To disable the service from starting automatically, you can type:

www-data@nine01-test:~ $ systemctl --user disable mymemcached.service
Removed symlink /home/www-data/.config/systemd/user/default.target.wants/mymemcached.service.

Unit Files

The unit files for the systemd user instance are stored in the directory ~/.config/systemd/user. The name of the file defines the name and type of the unit. For example, the file ~/.config/systemd/user/mymemcached.service defines a service unit named "mymemcached".

The unit description consist of three parts.

The Unit section contains metadata about the unit. The second section is named after the unit type, where Service is the most common one. This section defines the unit. The Install section contains instructions about how to activate / deactivate a unit.

To ensure a service is started during the system start, it has to be added to the default.target. Note this is not the often used multi-user.target as it is not available in the user instance of systemd.

Simple Service

This example starts a memcached server on port 11212.

The Unit section only defines the description and an (optional) URL for documentation purposes. The Service section defines the start and stop command and parameters. The stop command is optional. If systemd stops a service, it will first execute the stop command and then kill all leftover processes of this unit.

# ~/.config/systemd/user/mymemcached.service
[Unit]
Description=Custom memcached on port 11212
Documentation=https://docs.nine.ch/a/Ws7sDLPoiTo

[Service]
ExecStart=/usr/bin/memcached -A -m 32 -p 11212 -u www-data -l 127.0.0.1 -P /home/www-data/memcached.pid

[Install]
WantedBy=default.target

Forking Service

Systemd recommends against services that daemonize themselves. If your application needs to daemonize, you need to reflect this in the unit configuration by setting the service Type to forking in the Service section. As part of this, it is recommended to define the PIDFile option to tell systemd which process is the main process.

For convenience, specifiers can be used to create more robust and portable unit files. For example, %h is replaced with the users home directory and %p with the name of the unit. More specifiers are documented in the systemd.unit manpage.

# ~/.config/systemd/user/mymemcached.service
[Unit]
Description=Custom memcached on port 11212
Documentation=https://docs.nine.ch/a/Ws7sDLPoiTo

[Service]
Type=forking
ExecStart=/usr/bin/memcached -A -m 32 -p 11212 -u www-data -l 127.0.0.1 -P %h/.%p.pid

[Install]
WantedBy=default.target

Template Service

Templates are a powerful tool of systemd. Units with names ending with an @ are considered a template unit. Configuring a template enables you to launch multiple similar instances of the same service with different ports or sockets. The part following the @ is then factored into the %i specifier for use in the unit file.

If you paste the following example into a file named mymemcached@.service, you can launch multiple instances of it.

# .config/systemd/user/mymemcached@.service
[Unit]
Description=Custom memcached on port %i
Documentation=https://docs.nine.ch/a/Ws7sDLPoiTo

[Service]
ExecStart=/usr/bin/memcached -A -m 32 -p %i -u www-data -l 127.0.0.1 -P %h/%p-%i.pid

[Install]
WantedBy=default.target

For example, mymemcached@7777 and mymemcached@7778 would start a memcached server on port 7777 and 7778 respectively.

systemctl --user enable mymemcached@7778.service
Created symlink /home/www-data/.config/systemd/user/default.target.wants/mymemcached@7778.service → /home/www-data/.config/systemd/user/mymemcached@.service.

www-data@nine01-test:~ $ systemctl --user start mymemcached@7778.service

www-data@nine01-test:~ $ systemctl --user status mymemcached@7778.service
● mymemcached@7778.service - Custom memcached on port 7778
   Loaded: loaded (/home/www-data/.config/systemd/user/mymemcached@.service; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2022-01-27 08:55:58 CET; 7s ago
     Docs: /a/Ws7sDLPoiTo
 Main PID: 108919 (memcached)
   CGroup: /user.slice/user-33.slice/user@33.service/mymemcached.slice/mymemcached@7778.service
           └─108919 /usr/bin/memcached -A -m 32 -p 7778 -u www-data -l 127.0.0.1 -P /home/www-data/memcached-7778.pid

Jan 27 08:55:58 nine01-test systemd[1634]: Started Custom memcached on port 7778.

Advanced Configuration Snippets

Pre- and post Start Tasks

If your service depends on some extra steps during the start or stop routine, you can use the ExecStart and ExecStop options in combination with the suffixes Pre or Post. The parameter can be repeated multiple times for multiple steps and actions to execute.

By prefixing your command with a -, you can tell systemd to ignore errors of this command. Systemd will abort the start process if you don`t specify this flag and an error happens during startup.

[Service]
ExecStartPre=/bin/mkdir -p /tmp/mytempdir
ExecStartPre=-/bin/false
ExecStartPost=/usr/bin/touch /tmp/mytempdir/started
ExecStopPost=/bin/rm -rf /tmp/mytempdir

Reload

If your service is capable of reloading its configuration, you can use the ExecReload parameter to specify how to trigger the reload.

To reload the service, systemd executes the given command. The $MAINPID variable is set to the main process id of the service.

[Service]
ExecReload=/bin/kill -HUP $MAINPID

Restart

With the Restart parameter, you can specify whether systemd should restart the unit if it exits unexpectedly. The default for Restart is no. Setting it to always will make systemd restart the process automatically.

[Service]
Restart=always

If you are not monitoring the service in any other way, it is highly recommended to set Restart=always. Although this does not prevent any interruption, the impact will be smaller.

Environment Variables

In addition to configuration files, applications can also be configured through environment variables. Systemd allows you to set a working directory (WorkingDirectory) as well as environment variables (Environment) within the unit configuration.

Additionally, you're able to specify a file that contains more environment variables (EnvironmentFile), which could be part of your deployments. If you want to use this option, the file has to exist for systemd to be able to start the service. Prefixing the declaration with a - will make systemd ignore the option if the file does not exist. Entries in the EnvironmentFile need to be in a KEY=VALUE format.

[Service]
WorkingDirectory=/home/www-data/servicedir
Environment=ENV=production
EnvironmentFile=-/home/www-data/servicedir/.env

Links

More information and options about units can be found in the systemd.unit manpage. For Service units, the systemd.service and systemd.exec manpages are a good starting point.