mix_systemd

This library generates a systemd unit file to manage an Elixir application.

At its heart, it's a mix task which reads information about the project from mix.exs plus optional library configuration in config/config.exs and generates systemd unit files using Eex templates.

The goal is that the project defaults will generate a good systemd unit file, and standard options support more specialized use cases. If you need more customization, you can check the local copy of the templates into source control and modify them (and patches are welcome).

It uses standard systemd functions and conventions to make your app a more "native" OS citizen, and takes advantage of systemd features to improve security and reliability.

While it can be used standalone, more advanced use cases require scripts from e.g. mix_deploy.

Installation

Add mix_systemd to the list of dependencies in mix.exs:

def deps do
  [
    {:mix_systemd, "~> 0.1.0"}
  ]
end

Usage

This library works similarly to Distillery. The init task copies template files into your project, then the generate task uses them to create the output files.

Run this command to initialize templates under the rel/templates/systemd directory:

MIX_ENV=prod systemd.init

Next, generate output files under _project/#{mix_env}/systemd/lib/systemd/system.

MIX_ENV=prod mix systemd.generate

Configuration

The library gets standard information in mix.exs, e.g. the app name and version, then calculates default values for its configuration parameters.

You can override these parameters using settings in config/config.exs, e.g.:

config :mix_systemd,
    app_user: "app",
    app_group: "app",
    base_dir: "/opt",
    env_vars: [
        "REPLACE_OS_VARS=true",
    ]

The following sections describe configuration options. See lib/mix/tasks/systemd.ex for all the details.

If you need to make changes not supported by the config options, then you can check the templates into source control from rel/templates/systemd and make your own changes (contributions welcome!).

Basics

app_name: Elixir application name, an atom, from the app field in the mix.exs project.

version: version from mix.exs project.

ext_name: External name, used for files and directories. Default is app_name with underscores converted to "-".

service_name: Name of the systemd service, default ext_name.

base_dir: Base directory where app files go, default /srv to follow conventions.

deploy_dir: Directory where app files go, default #{base_dir}/#{ext_name}

app_user: OS user account that the app runs under, default ext_name.

app_group: OS group account, default ext_name.

Directories

Modern Linux defines a set of directories which apps use for common purposes, e.g. configuration or cache files. See https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RuntimeDirectory=

This library defines these directories based on the app name, e.g. /etc/#{ext_name}. It only creates directories that the app uses, default runtime (/run/#{ext_name}) and configuration (/etc/#{ext_name}). If your app uses other dirs, set them in the dirs var:

dirs: [
  :runtime,       # Needed for RELEASE_MUTABLE_DIR, runtime-environment or conform
  :configuration, # Needed for Erlang cookie
  # :cache,       # App cache files which can be deleted
  # :logs,        # App external log files, not via journald
  # :state,       # App state persisted between runs
  # :tmp,         # App temp files
],

For security, we set permissions to 750, more restrictive than the systemd defaults of 755. You can configure them with e.g. configuration_directory_mode. See the defaults in lib/mix/tasks/systemd.ex.

More recent versions of systemd (after 235) will create these directories at start time based on the settings in the unit file. For earlier systemd versions, you need to create them beforehand using scripts in e.g. mix_deploy.

systemd_version: Sets the systemd version on the target system, default 235. This determines which systemd features the library will enable. If you are targeting an older OS relese, you may need to change it. Here are the systemd versions in common OS releases:

  • CentOS 7: 219
  • Ubuntu 16.04: 229
  • Ubuntu 18.04: 237

Additional directories

The library assues a directory structure under deploy_dir which allows it to handle multiple reases, similar to Capistrano.

  • scripts_dir: dir for deployment scripts which e.g. start and stop the unit, default bin.
  • current_dir: dir where the current Erlang release is unpacked or referenced by symlink, default current.
  • releases_dir: dir where versioned releases may be unpacked, default releases.
  • flags_dir: dir for flag files to trigger restart, e.g. when restart_method is :systemd_flag, default flags.

When using multiple releases and symlinks, the deployment process works like this:

  1. Create a new directory for the release with a timestamp like /srv/foo/releases/20181114T072116.

  2. Upload the new release tarball to the server and unpack it to the release dir

  3. Make a symlink from /srv/foo/current to the new release dir.

  4. Restart the app.

If you are only keeping a single version, then you would deploy it to the /srv/foo/current dir.

Environment vars

The library sets a few common env vars directly in the unit file:

  • PORT: env_port var, default 4000
  • LANG: env_lang var, default en_US.UTF-8
  • MIX_ENV: mix_env var, default Mix.env()
  • HOME: home_dir var, default deploy_dir
  • RELEASE_MUTABLE_DIR: default runtime_dir, e.g. /run/#{ext_name}
  • DEFAULT_COOKIE_FILE: default #{configuration_dir}/erlang.cookie, e.g. /etc/#{ext_name}/erlang.cookie
  • CONFORM_CONF_PATH: default /etc/#{ext_name}/#{app_name}.conf. if conform is true

You can set additional vars in the env_vars config var, e.g.:

env_vars: [
    "REPLACE_OS_VARS=true",
]

The unit file also attempts to read environment vars from a series of files:

  • etc/environment within the release, e.g. /srv/app/currrent/etc/environment
  • #{deploy_dir}/etc/environment, e.g. /srv/app/etc/environment
  • #{configuration_dir}/environment, e.g. /etc/app/environment
  • #{runtime_dir}/runtime-environment, e.g. /var/run/app/runtime-environment

Later values override earlier values, so you can set defaults which get overridden in the deployment or runtime environment.

Systemd and OS

limit_nofile, Limit on open files, systemd LimitNOFILE, default 65535.

umask, process umask, systemd UMask, default 0027

restart_sec: time to wait between restarts, systemd RestartSec, default 5 sec.

service_type: :simple | :exec | :notify | :forking. Default :forking.

In theory, "modern" applications are not supposed to fork. The Erlang VM runs pretty well in "foreground" mode, but it is really expecting to run as a standard Unix-style daemon. Systemd expects foregrounded apps to die when their pipe closes, so this library runs forking mode by default. It sets pid_file to runtime_directory/#{app_name}.pid and sets the PIDFILE env var to tell Distillery where it is. See https://elixirforum.com/t/systemd-cant-shutdown-my-foreground-app-cleanly/14581/2

To run in foreground mode, set service_type to :simple or :exec. Note that in simple mode, systemd doesn't actually check if the app started successfully, it just keeps going. If something depends on your app being up, :exec may be better.

restart_method: :systemd_flag | :systemctl | :touch. Default :systemctl

Set this to :systemd_flag, and the library will generate an additional unit file which watches for changes to a flag file and restarts the main unit. This allows updates to be pushed to the target machine by an unprivilieged user account which does not have permissions to restart proccesses. touch the file #{flags_dir}/restart.flag and systemd will restart the unit.

Runtime configuration

For configuration, we normally use a combination of build time settings, deploy time settings, and runtime settings.

The configuration settings in config/prod.exs are baked into the release. We can then extend them with machine specific configuration stored in the configuration dir /etc/#{ext_name} which are read by the app on startup.

In on-premises deployments, we might generate the machine-specific configuration when setting up the app. In cloud and other dynamic environments, we may run from a read-only image, e.g. an Amazon AMI, which gets configured for the environment when it starts.

We can also read configuration settings at runtime, e.g. getting the database host and login from a configuration store like AWS Systems Manager Parameter Store or etcd.

We might also have things that change dynamically, e.g. the IP address of the machine.

Conform is a popular way of making a machine-specific config file. Set conform to true, and the library will set CONFORM_CONF_PATH to /etc/#{ext_name}/#{app_name}.conf. Conform has been depreciated in favor of TOML, so you should use that instead.

This library suports three ways to get runtime config:

ExecStartPre scripts

These scripts run before the main ExecStart script runs.

You can specify multiple scripts in the exec_start_pre var. If the name starts with a slash, it is run directly, otherwise it is expected to be in the directory specified by scripts_dir (normally bin in the deploy dir).

These scripts should generally write config to either the systemd configuration_dir (/etc) for persistent config, and under runtime_dir (/run) for data which systemd should clean up on restart.

Wrapper script

Instead of running the main ExecStart script directly, run a shell script which sets up the environment, then exec the main script.

This is most useful for things that are truly dynamic and may change on restart. For example, if the app is in a cluster, we might get the IP address of the primary network interface to set the node name.

Set runtime_environment_wrap to true and set runtime_environment_wrap_script to the name of the script. Default is the deploy-runtime-environment-wrap script from mix_deploy.

Systemd starts units in parallel when possible, but we may need to enforce ordering. Set runtime_environment_service_after to the names of systemd units that the script depends on. For example, set it to cloud-init.target if you are using cloud-init to get runtime network information.

Systemd service

We can run our own service to collect runtime data and configure the system. Set runtime_environment_service to true and this library will create a service which runs the script specified by runtime_environment_service_script and make it a runtime dependency of the main script.

Security

paranoia: enable systemd security options, default false.

NoNewPrivileges=yes
PrivateDevices=yes
PrivateTmp=yes
ProtectSystem=full
ProtectHome=yes
PrivateUsers=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectControlGroups=yes
MountAPIVFS=yes
                                                                                                

chroot: Enable systemd chroot, default false. Systemd RootDirectory is set to current_dir. You can also set systemd ReadWritePaths=, ReadOnlyPaths=, InaccessiblePaths= with the read_write_paths, read_only_paths and inaccessible_paths vars.