mix_systemd
This library generates a systemd unit file to manage an Elixir release. It supports releases generated by Elixir 1.9+ mix release or Distillery.
At its heart, it's a mix task which reads information about the project from
mix.exs
and config/config.exs
then generates systemd unit files using Eex
templates. The goal is that the project defaults will generate a good systemd
unit file, and standard options support more specialized use cases.
It uses standard systemd functions and conventions to make your app a more "native" OS citizen, and takes advantage of systemd features to improve security and reliability. While it can be used standalone, more advanced use cases use scripts from e.g. mix_deploy.
This complete example app puts the pieces together.
Installation
Add mix_systemd
to the list of dependencies in mix.exs
:
def deps do
[
{:mix_systemd, "~> 0.7"},
]
end
Configuration
The library tries to choose reasonable defaults, so you may not need to
configure anything. It reads the app name from mix.exs
and calculates default
values for its configuration parameters. For example, if your app is named
foo_bar
, it will create a service named foo-bar
, deployed to
/srv/foo-bar
, running under the user foo-bar
.
You can override these parameters using settings in config/config.exs
, e.g.:
config :mix_systemd,
app_user: "app",
app_group: "app",
base_dir: "/opt",
env_vars: [
"PORT=8080",
]
Configuration strategies
There are four different kinds of things that we may want to configure:
Static information about application layout, e.g. file paths. This is the same for all machines in an environment, e.g. staging or prod.
Information specific to the environment, e.g. the hostname of the db server.
Secrets such as db passwords, API keys or encryption keys.
Dynamic information such as the IP address of the server or other other machines in the cluster.
Elixir has a couple of mechanisms for storing configuration. When you compile
the release, it converts Elixir-format config files like config/config.exs
into an initial application environment that is read by Application.get_env/3
.
That's good for simple, relatively static apps. It's not ideal to store
passwords in the release file, though.
Elixir 1.9 releases support dynamic configuration at runtime. You can run the
Elixir file config/releases.exs
when it boots or use the shell script
rel/env.sh.eex
to set environment vars. With these you can theoretically do
anything. In practice, however, it can be more convenient and secure to process
the config outside of the app. That's where mix_systemd
and mix_deploy
come
in.
Beyond that, the configuration process depends on how complex things are.
Environment vars
The simplest thing is to set environment variables. Add individual vars to
env_vars
, and they will be set in the systemd unit file. Add files to
env_files
and systemd will load them before starting your app.
Your application then calls System.get_env/1
in config/releases.exs
or
application startup. Note that these environment vars are read at runtime,
not when building your app.
env_vars: [
# Set a variable, good for things that are not sensitive and don't change
"PORT=8080",
],
dirs: [
# create /etc/foo
:configuration,
],
env_files: [
# Read environment vars from the file /etc/foo/environment
["-", :configuration_dir, "/environment"],
]
/etc/foo/environment
looks like:
DATABASE_URL="ecto://foo_prod:Sekrit!@db.foo.local/foo_prod"
SECRET_KEY_BASE="EOdJB1T39E5Cdeebyc8naNrOO4HBoyfdzkDy2I8Cxiq4mLvIQ/0tK12AK1ahrV4y"
HOST="www.example.com"
ASSETS_HOST="assets.example.com"
RELEASE_COOKIE="LmCMGNz04yEJ4MQc6jt3cS7QjAppYOw_bQa7NE5hPZJGqL3Yry1jUg=="
config/releases.exs
looks like:
config :foo, Foo.Repo,
url: System.get_env("DATABASE_URL")
config :foo, FooWeb.Endpoint,
http: [:inet6, port: System.get_env("PORT") || 4000],
url: [host: System.get_env("HOST"), port: 443],
static_url: [host: System.get_env("ASSETS_HOST"), port: 443],
secret_key_base: System.get_env("SECRET_KEY_BASE"),
cache_static_manifest: "priv/static/cache_manifest.json"
Copying files
The question is how to get the environment files onto the server. For simple server deployments, we can copy the config to the server when doing the initial setup. In cloud environments, we may run from a read-only image, e.g. an Amazon AMI, which gets configured at start up based on the environment by copying the config from an S3 bucket.
umask 077
aws s3 sync --exact-timestamps --no-progress "s3://${CONFIG_BUCKET}/" "/etc/foo/"
chown -R $DEPLOY_USER:$APP_GROUP /etc/foo
find /etc/foo -type f -exec chmod 640 {} \;
find /etc/foo -type d -exec chmod 750 {} \;
The following example runs the script /srv/foo/bin/deploy-sync-config-s3
from
mix_deploy
. It uses an environment file in /srv/foo/etc/environment
to bootstrap the sync, e.g. setting the S3 bucket name. That file
is placed there by CodeDeploy at deploy time.
config :mix_systemd,
exec_start_pre: [
# Run before starting the app
# The `!` means the script is run as root, not as the app user
["!", :deploy_dir, "/bin/deploy-sync-config-s3"]
],
dirs: [
:configuration, # /etc/foo, app configuration, e.g. db passwords
:runtime, # /run/foo, temp files which may be deleted between runs
],
# runtime_directory_preserve: "yes", # don't delete dir for debugging
env_files: [
["-", :deploy_dir, "/etc/environment"], # /srv/foo/etc/environment
]
env_vars: [
# Tell release to use /run/foo for temp files
["RELEASE_TMP=", :runtime_dir],
]
Config providers
At a certain point, making everything into an environment var becomes annoying. It's verbose and vars are simple strings, so you have to encode values safely and convert them back to lists, integers or atoms.
Config providers let you load files in standard formats like TOML.
[foo."Foo.Repo"]
url = "ecto://foo_prod:Sekrit!@db.foo.local/foo_prod"
pool_size = 15
[foo."FooWeb.Endpoint"]
secret_key_base = "EOdJB1T39E5Cdeebyc8naNrOO4HBoyfdzkDy2I8Cxiq4mLvIQ/0tK12AK1ahrV4y"
The app reads these config files on startup and merges them into the app config.
defp releases do
[
prod: [
include_executables_for: [:unix],
config_providers: [
{TomlConfigProvider, path: "/etc/foo/config.toml"}
],
steps: [:assemble, :tar]
]
]
end
The startup scripts read the initial application environment compiled into the
release, parse the config file, merge the values, write it to a temp file then
start the VM. Because of that, they need a writable directory. That is
configured using the RELEASE_TMP
environment var, normally set to the app's
runtime_dir
.
dirs: [
:configuration,
:runtime,
],
env_vars: [
["RELEASE_TMP=", :runtime_dir],
],
Config servers and vaults
You can also store config params in an external configuration system and read them at runtime. An example is AWS Systems Manager Parameter Store.
Set a parameter using the AWS CLI:
aws ssm put-parameter --name '/foo/prod/db/password' --type ‘SecureString’ --value 'Sekrit!"
While it's possible to read params in config/releases.exs
, it's tedious.
Better is to grab all of them at once and write them to a file, then read it in
with a Config Provider like aws_ssm_provider.
aws --region us-east-1 ssm get-parameters-by-path --path "/foo/prod/" --recursive --with-decryption --query "Parameters[]" > /etc/foo/ssm.json
defp releases do
[
prod: [
include_executables_for: [:unix],
config_providers: [
{AwsSsmProvider, path: "/etc/foo/ssm.json"}
],
steps: [:assemble, :tar]
]
]
end
Dynamic config
You can write code to do things like query the system for the primary IP
address, but cloud-init
already does it,
you just have to read the JSON file
The most common use for this is setting up the VM node name. In env.sh
:
CLOUD_NAME=$(jq -r '.v1.cloud_name' < /run/cloud-init/instance-data.json)
if [ "$CLOUD_NAME" = "digitalocean" ]; then
IP_ADDR=$(jq -r '.ds.meta_data.interfaces.public[0].anchor_ipv4.ip_address' < /run/cloud-init/instance-data.json)
DEFAULT_IPV4="$IP_ADDR"
elif [ "$CLOUD_NAME" = "aws" ]; then
IP_ADDR=$(jq -r '.ds.meta_data."local-ipv4"' < /run/cloud-init/instance-data.json)
# IP_ADDR=$(jq -r '.ds.meta_data."public-ipv4"' < /run/cloud-init/instance-data.json)
AWS_REGION=$(jq -r '.v1.region' < /run/cloud-init/instance-data.json)
fi
RELEASE_DISTRIBUTION="name"
RELEASE_NODE="${RELEASE_NAME}@${IP_ADDR}"
Security
An important security principle is "least privilege". If an attacker manages to compromise the app, then they can do whatever it has permissions to do, not just what you expect. Because of that, I prefer that the account that the app runs under cannot write files, and having a writable config file that is also executed is the worst case scenario.
Usage
First, use the systemd.init
task to template files from the library to the
rel/templates/systemd
directory in your project.
mix systemd.init
Next, generate output files in the build directory under
_build/#{mix_env}/systemd/lib/systemd/system
.
MIX_ENV=prod mix systemd.generate
Configuration options
The following sections describe common configuration options.
See lib/mix/tasks/systemd.ex
for the details of more obscure options.
If you need to make changes not supported by the config options,
then you can check the templates in rel/templates/systemd
into source control and make your own changes. Contributions are welcome!
Basics
app_name
: Elixir application name, an atom, from the app
field in the mix.exs
project.
module_name
: Elixir camel case module name version of app_name
, e.g. FooBar
.
release_name
: Name of release, default app_name
.
ext_name
: External name, used for files and directories,
default app_name
with underscores converted to "-", e.g. foo-bar
.
service_name
: Name of the systemd service, default ext_name
.
release_system
: :mix | :distillery
, default :mix
Identifies the system used to generate the releases, Mix or Distillery.
Users
app_user
: OS user account that the app runs under, default ext_name
.
app_group
: OS group account, default ext_name
.
Directories
base_dir
: Base directory for app files on target, default /srv
.
deploy_dir
: Directory for app files on target, default #{base_dir}/#{ext_name}
.
We use the
standard app directories,
for modern Linux systems. App files are under /srv
, configuration under
/etc
, transient files under /run
, data under /var/lib
.
Directories are named based on the app name, e.g. /etc/#{ext_name}
.
The dirs
variable specifies which directories the app uses.
By default, it doesn't set up anything. To enable them, configure the dirs
param, e.g.:
dirs: [
# :runtime, # App runtime files which may be deleted between runs, /run/#{ext_name}
# :configuration, # App configuration, e.g. db passwords, /etc/#{ext_name}
# :state, # App data or state persisted between runs, /var/lib/#{ext_name}
# :cache, # App cache files which can be deleted, /var/cache/#{ext_name}
# :logs, # App external log files, not via journald, /var/log/#{ext_name}
# :tmp, # App temp files, /var/tmp/#{ext_name}
],
Recent versions of systemd (since 235) will create these directories at start time based on the settings in the unit file. With earlier systemd versions, create them beforehand using installation scripts, e.g. mix_deploy.
For security, we set permissions to 750, more restrictive than the systemd
defaults of 755. You can configure them with variables like
configuration_directory_mode
. See the defaults in
lib/mix/tasks/systemd.ex
.
systemd_version
: Sets the systemd version on the target system, default 235.
This determines which systemd features the library will enable. If you are
targeting an older OS release, you may need to change it. Here are the systemd
versions in common OS releases:
- CentOS 7: 219
- Ubuntu 16.04: 229
- Ubuntu 18.04: 237
Additional directories
The library uses a directory structure under deploy_dir
which supports
multiple releases, similar to Capistrano.
scripts_dir
: deployment scripts which e.g. start and stop the unit, defaultbin
.current_dir
: where the current Erlang release is unpacked or referenced by symlink, defaultcurrent
.releases_dir
: where versioned releases are unpacked, defaultreleases
.flags_dir
: dir for flag files to trigger restart, e.g. whenrestart_method
is:systemd_flag
, defaultflags
.
When using multiple releases and symlinks, the deployment process works as follows:
Create a new directory for the release with a timestamp like
/srv/foo/releases/20181114T072116
.Upload the new release tarball to the server and unpack it to the releases dir
Make a symlink from
/srv/#{ext_name}/current
to the new release dir.Restart the app.
If you are only keeping a single version, then deploy it to the directory
/srv/#{ext_name}/current
.
Variable expansion
The following variables support variable expansion:
expand_keys: [
:env_files,
:env_vars,
:runtime_environment_service_script,
:exec_start_pre,
:exec_start_wrap,
:read_write_paths,
:read_only_paths,
:inaccessible_paths,
]
You can specify values as a list of terms, and it will look up atoms as keys in
the config. This lets you reference e.g. the deploy dir or configuration dir without
having to specify the full path, e.g. ["!", :deploy_dir, "/bin/myscript"]
gets
converted to "!/srv/foo/bin/myscript"
.
Environment vars
The library sets env vars in the unit file:
MIX_ENV
:mix_env
, defaultMix.env()
LANG
:env_lang
, defaulten_US.utf8 *
RUNTIME_DIR:
runtime_dir, if
:runtimein
dirs*
CONFIGURATION_DIR:
configuration_dir, if
:configurationin
dirs*
LOGS_DIR:
logs_dir, if
:logsin
dirs*
CACHE_DIR:
cache_dir, if
:cachein
dirs*
STATE_DIR:
state_dir, if
:statein
dirs*
TMP_DIR:
tmp_dir, if
:tmpin
dirsYou can set additional vars using
env_vars, e.g.: ```elixir env_vars: [ "PORT=8080", ] ``` You can also reference the value of other parameters by name, e.g.: ```elixir env_vars: [ ["RELEASE_TMP=", :runtime_dir], ] ``` You can read environment vars from files with
env_files, e.g.: ```elixir env_files: [ ["-", :deploy_dir, "/etc/environment"], ["-", :configuration_dir, "environment"], ["-", :runtime_dir, "environment"], ], ``` The "-" at the beginning makes the file optional, the system will start without them. Later values override earlier values, so you can set defaults in the release which get overridden in the deployment or runtime environment. ### Runtime dirs The release scripts may need to write temp files and log files, e.g. when generating the application config config files. By default, they do this under the release dir, e.g.
/srv/foo/current/tmp. For security, it's better to deploy the app using a different user account from the one that the app runs under, making the source files read only. This makes it harder for an attacker to be able to make changes to the source and then make the app run them. In that case, we need to set an environment var which tells the release startup scripts where they can write files. For Mix releases, that is
RELEASE_TMPand for Distillery it is
RELEASE_MUTABLE_DIR, e.g.: ```elixir env_vars: [ {"RELEASE_TMP=", :runtime_dir}, ] ``` By default systemd will delete the runtime directory when restarting the app. This can be annoying when debugging startup issues. You can set
runtime_directory_preserveto
restartor
yes(see [RuntimeDirectoryPreserve](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RuntimeDirectoryPreserve=)). ### Starting and restarting The following variables set systemd variables:
service_type:
:simple | :exec | :notify | :forking. systemd [Type](https://www.freedesktop.org/software/systemd/man/systemd.service.html#Type=), default
:simple. Modern applications don't fork, they run in the foreground and rely on the supervisor to manage them as a daemon. This is done by setting
service_typeto
:simpleor
:exec. Note that in
simplemode, systemd doesn't actually check if the app started successfully, it just continues starting other units. If something depends on your app being up,
:execmay be better. Set
service_typeto
:forking, and the library sets
pid_fileto
#{runtime_directory}/#{app_name}.pidand sets the
PIDFILEenv var to tell the boot scripts where it is. The Erlang VM runs pretty well in foreground mode, but traditionally runs as as a standard Unix-style daemon, so forking might be better. Systemd expects foregrounded apps to die when their pipe closes. See https://elixirforum.com/t/systemd-cant-shutdown-my-foreground-app-cleanly/14581/2
restart_method:
:systemctl | :systemd_flag | :touch, default
:systemctlSet this to
:systemd_flag, and the library will generate an additional unit file which watches for changes to a flag file and restarts the main unit. This allows updates to be pushed to the target machine by an unprivileged user account which does not have permissions to restart processes. Touch the file
#{flags_dir}/restart.flagand systemd will restart the unit.
working_dir: Current working dir for app. systemd [WorkingDirectory](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#WorkingDirectory=), default
current_dir.
limit_nofile: Limit on open files, systemd [LimitNOFILE](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#LimitCPU=), default 65535.
umask: Process umask, systemd [UMask](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#UMask=), default "0027". Note that this is octal, so it needs to be a string.
restart_sec: Time in seconds to wait between restarts, systemd [RestartSec](https://www.freedesktop.org/software/systemd/man/systemd.service.html#RestartSec=), default 100ms.
syslog_identifier: Logging name, systemd [SyslogIdentifier](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#SyslogIdentifier=), default
service_name##
ExecStartPrescripts Scripts specified in
exec_start_pre(systemd [ExecStartPre](https://www.freedesktop.org/software/systemd/man/systemd.service.html#ExecStartPre=)]) run before the main
ExecStartscript runs, e.g.: ```elixir exec_start_pre: [ ["!", :deploy_dir, "/bin/deploy-sync-config-s3"] ] ``` This runs the
deploy-sync-config-s3script from
mix_deploy, which copies config files from an S3 bucket into
/etc/foo. By default, scripts run as the same user and group as the main script. Putting
!in front makes the script run with [elevated privileges](https://www.freedesktop.org/software/systemd/man/systemd.service.html#ExecStart=), allowing it to write config to
/etc/fooeven if the main user account cannot for security reasons. #### ExecStart wrapper script Instead of running the main
ExecStartscript directly, you can run a shell script which sets up the environment, then runs the main script with
exec. Set
exec_start_wrapto the name of the script, e.g.
deploy-runtime-environment-wrapfrom
mix_deploy. In Elixir 1.9+ releases you can use
env.sh, but this runs earlier with elevated permissions, so it may be useful as well. #### Runtime environment service You can run your own separate service to configure the runtime environment before the app runs. Set
runtime_environment_service_scriptto a script such as
deploy-runtime-environment-filefrom
mix_deploy. This library will create a
#{service_name}-runtime-environment.serviceunit and make it a systemd runtime dependency of the app. ### Runtime dependencies Systemd starts units in parallel when possible. To enforce ordering, set
unit_after_targetsto the names of systemd units that this unit depends on. For example, if this unit should run after cloud-init to get [runtime network information](https://cloudinit.readthedocs.io/en/latest/topics/network-config.html#network-configuration-outputs), set: ```elixir unit_after_targets: [ "cloud-init.target" ] ``` ## Security
paranoia: Enable systemd security options, default
false. NoNewPrivileges=yes PrivateDevices=yes PrivateTmp=yes ProtectSystem=full ProtectHome=yes PrivateUsers=yes ProtectKernelModules=yes ProtectKernelTunables=yes ProtectControlGroups=yes MountAPIVFS=yes │
chroot: Enable systemd [chroot](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RootDirectory=), default
false. Sets systemd
RootDirectoryis set to
current_dir. You can also set systemd [ReadWritePaths=, ReadOnlyPaths=, InaccessiblePaths=](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#ReadWritePaths=) with the
read_write_paths,
read_only_pathsand
inaccessible_pathsvars, respectively. ## Distillery Distillery has largely been replaced by Elixir native releases. This library works fine with it, though.
exec_start_prescripts are particularly useful in the absense of
env.sh. Configure the library by setting
release_system: :distillery, e.g.. ```elixir config :mix_systemd, release_system: :distillery, exec_start_pre: [ # Run script as root before starting ["!", :deploy_dir, "/bin/deploy-sync-config-s3"] ], dirs: [ :configuration, :runtime, ], runtime_directory_preserve: "yes", env_vars: [ # Use /run/foo for temp files ["RELEASE_MUTABLE_DIR=", :runtime_dir], # expand $CONFIGURATION_DIR in config files REPLACE_OS_VARS=true, ] ``` Set up [config providers](https://hexdocs.pm/distillery/Mix.Releases.Config.Providers.Elixir.html) in
rel/config.exs: ```elixir environment :prod do set config_providers: [ {Mix.Releases.Config.Providers.Elixir, ["${CONFIGURATION_DIR}/config.exs"]} ] end ``` This reads files in Elixir config format. Instead of including your
prod.secret.exsfile in
prod.exs, you can copy it to the server separately, and it will be read at startup. The [TOML configuration provider](https://github.com/bitwalker/toml-elixir) works similarly: ```elixir environment :prod do set config_providers: [ {Toml.Provider, [path: "${CONFIGURATION_DIR}/config.toml"]}, ] end ``` Add the TOML config provider to
mix.exs: ```elixir {:toml_config_provider, "~> 0.2.0"} ``` You can generate a file under the release with an overlay in
rel/config.exs`, e.g.:elixir environment :prod do set overlays: [ {:mkdir, "etc"}, {:copy, "rel/etc/environment", "etc/environment"}, # {:template, "rel/etc/environment", "etc/environment"} ] end
That results in a file that would be read by:elixir env_files: [ ["-", :current_dir, "/etc/environment"], ],