Security

This guide covers FerricStore server security: protected mode, ACL users, passwords, TLS, Erlang distribution, audit logs, and production limits.

Local examples may disable protected mode for convenience. Production deployments should use protected mode, ACL, TLS where appropriate, private network binding, and a strong Erlang cookie for clustered nodes.

FerricStore provides multiple layers of security: ACL user accounts with fine-grained permissions, TLS encryption, protected mode, password hashing, and audit logging.

Access Control Lists (ACL)

FerricStore implements native-protocol ACL with enhancements for production use.

Default User

On startup, a default user is created with:

• Enabled
• No password required
• Full access to all commands (+@all)
• Full access to all keys (~*)

The default user cannot be deleted.

Creating Users

# Create a read-only user with password
ACL SETUSER reader on >secretpassword +@read ~*

# Create an admin user with full access
ACL SETUSER admin on >adminpassword +@all ~*

# Create a user restricted to specific key patterns
ACL SETUSER appuser on >apppassword +@all ~app:*

# Create a user with specific commands only
ACL SETUSER metrics on >metricspass +INFO +DBSIZE +KEYS ~*

ACL Rules

Command Categories

Authenticating

# With default user (if password is set)
AUTH mypassword

# With named user
AUTH username password

Managing Users

# List all users
ACL LIST

# Get user details
ACL GETUSER reader

# Check current user
ACL WHOAMI

# Delete a user
ACL DELUSER reader

Persisting ACL

# Save current ACL to data_dir/acl.conf
ACL SAVE

# Reload ACL from file
ACL LOAD

ACL state is auto-loaded from data_dir/acl.conf on startup if the file exists. The file is written atomically (tmp + fsync + rename) with 0600 permissions.

Password Hashing

Passwords are never stored in plaintext. FerricStore uses PBKDF2-SHA256 with:

• 100,000 iterations
• Random salt per password
• :crypto module for cryptographic operations

When you set a password with >password, the plaintext is immediately hashed and only the hash is stored in the ACL ETS table. New hashes use a versioned pbkdf2-sha256$... format. Existing unversioned PBKDF2 hashes remain accepted for migration and are upgraded in memory after a successful authentication; the legacy fast SHA-256 fallback is not accepted. ACL LOAD rejects files containing plaintext passwords.

Protected Mode

When FerricStore starts with the default user having no password and no other ACL users configured, protected mode is active. In this state, non-localhost connections are rejected.

To disable protected mode, either:

1. Set a password on the default user: ACL SETUSER default on >password
2. Create at least one non-default user with a password

Dashboard Bootstrap and Login

When FERRICSTORE_PROTECTED_MODE=true, the OSS dashboard uses the same durable ACL identities and permissions as the native protocol. It does not maintain a separate user database.

A new installation whose replicated ACL catalog contains only the initial, passwordless default user redirects dashboard HTML requests to /dashboard/setup. The setup form sets a confirmed password on default, creates a signed dashboard session, and then becomes unavailable. The catalog compare-and-swap makes concurrent setup attempts deterministic: only the first valid commit succeeds. Setup is exposed only for the canonical factory ACL state, so it cannot overwrite an existing administrator. Deliberately restoring the durable ACL catalog to that factory state re-enables bootstrap.

For a dashboard reached directly from loopback, setup does not require a second credential. Remote setup requires all of the following:

• FERRICSTORE_DASHBOARD_REMOTE_ACCESS=true
• HTTPS termination at an explicitly trusted proxy
• FERRICSTORE_DASHBOARD_TRUST_PROXY_HEADERS=true
• the proxy address or CIDR in FERRICSTORE_DASHBOARD_TRUSTED_PROXIES
• FERRICSTORE_DASHBOARD_SESSION_SECRET set to at least 32 random bytes
• FERRICSTORE_DASHBOARD_BOOTSTRAP_TOKEN_FILE pointing to a mounted file that contains a random 32-4096 byte bootstrap credential

For example:

umask 077
openssl rand -hex 32 > /run/secrets/ferricstore-dashboard-bootstrap
export FERRICSTORE_DASHBOARD_REMOTE_ACCESS=true
export FERRICSTORE_DASHBOARD_TRUST_PROXY_HEADERS=true
export FERRICSTORE_DASHBOARD_TRUSTED_PROXIES=10.0.0.0/8
export FERRICSTORE_DASHBOARD_SESSION_SECRET="$(openssl rand -hex 32)"
export FERRICSTORE_DASHBOARD_BOOTSTRAP_TOKEN_FILE=/run/secrets/ferricstore-dashboard-bootstrap

All dashboard-serving nodes behind the same endpoint must use the same session secret. Mount the same bootstrap token on every node that may receive the initial setup request.

Do not enable FERRICSTORE_DASHBOARD_ALLOW_INSECURE_HTTP on an untrusted network. Forwarded scheme and client-address headers are ignored unless the immediate peer is in the trusted proxy list.

After bootstrap, use Security > Account management to add named accounts:

The account-management surface itself follows ACL policy. Listing requires ACL.LIST, account creation, state, password, and rule changes require ACL.SETUSER, and deletion requires ACL.DELUSER. The default recovery administrator cannot be disabled or deleted in the dashboard, and a user cannot disable or delete its own active account there. Password or rule changes invalidate existing signed sessions immediately.

Dashboard POSTs use origin validation and double-submit CSRF tokens. Login and bootstrap attempts share the bounded authentication rate limiter, and authentication plus dashboard ACL mutations are available to the audit log.

TLS Configuration

FerricStore has two separate network layers, each with its own TLS:

Configuring client TLS does not encrypt Raft traffic between nodes. In production multi-node deployments, you should configure both.

Client-Server TLS

Encrypt all client-server communication with TLS.

Server Configuration

# config/config.exs
config :ferricstore, :native_tls_port, 6389
config :ferricstore, :native_tls_cert_file, "/etc/ssl/ferricstore/server-cert.pem"
config :ferricstore, :native_tls_key_file, "/etc/ssl/ferricstore/server-key.pem"

Mutual TLS (Client Certificate Authentication)

config :ferricstore, :native_tls_ca_cert_file, "/etc/ssl/ferricstore/ca-cert.pem"

When native_tls_ca_cert_file is set, the server requires clients to present a certificate signed by the specified CA.

Enforcing TLS Only

config :ferricstore, :require_tls, true

When require_tls is true, the plaintext TCP listener still starts (for health checks, local admin) but rejects non-TLS client connections.

Connecting with TLS

Native SDK clients should connect to :native_tls_port, validate the server certificate with the configured CA, and present a client certificate when mTLS is enabled. Exact client options are SDK-specific, but the server-side contract is always the native protocol over TLS on the configured TLS port.

TLS in Kubernetes

In Kubernetes, use cert-manager to issue and rotate certificates automatically. Mount them as a Secret volume into the FerricStore pod.

1. Create a Certificate resource:

# ferricstore-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ferricstore-tls
  namespace: default
spec:
  secretName: ferricstore-tls-secret
  duration: 2160h    # 90 days
  renewBefore: 360h  # renew 15 days before expiry
  issuerRef:
    name: my-cluster-issuer   # your cert-manager ClusterIssuer
    kind: ClusterIssuer
  dnsNames:
    - ferricstore
    - ferricstore.default.svc.cluster.local
    - "*.ferricstore-headless.default.svc.cluster.local"  # for multi-node

2. Mount the secret in the pod:

# ferricstore-deployment.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ferricstore
spec:
  template:
    spec:
      containers:
        - name: ferricstore
          env:
            - name: TLS_CERT_FILE
              value: /etc/tls/tls.crt
            - name: TLS_KEY_FILE
              value: /etc/tls/tls.key
            - name: TLS_CA_FILE
              value: /etc/tls/ca.crt
          volumeMounts:
            - name: tls-certs
              mountPath: /etc/tls
              readOnly: true
      volumes:
        - name: tls-certs
          secret:
            secretName: ferricstore-tls-secret

3. Configure FerricStore to read from the mounted paths:

# config/runtime.exs
config :ferricstore, :native_tls_port, 6389
config :ferricstore, :native_tls_cert_file, System.get_env("TLS_CERT_FILE")
config :ferricstore, :native_tls_key_file, System.get_env("TLS_KEY_FILE")
config :ferricstore, :native_tls_ca_cert_file, System.get_env("TLS_CA_FILE")
config :ferricstore, :require_tls, true

cert-manager automatically renews the certificate and updates the Secret. Kubernetes restarts the pod when the secret changes (or use a sidecar like reloader to trigger a graceful restart).

TLS in Nomad

In Nomad, use Vault's PKI secrets engine for certificate issuance.

1. Vault template in the job spec:

job "ferricstore" {
  group "cache" {
    task "ferricstore" {
      template {
        data = <<EOF
{{ with secret "pki/issue/ferricstore" "common_name=ferricstore.service.consul" "ttl=720h" }}
{{ .Data.certificate }}
{{ end }}
EOF
        destination = "secrets/server-cert.pem"
        change_mode = "restart"
      }

      template {
        data = <<EOF
{{ with secret "pki/issue/ferricstore" "common_name=ferricstore.service.consul" "ttl=720h" }}
{{ .Data.private_key }}
{{ end }}
EOF
        destination = "secrets/server-key.pem"
        change_mode = "restart"
      }

      template {
        data = <<EOF
{{ with secret "pki/issue/ferricstore" "common_name=ferricstore.service.consul" "ttl=720h" }}
{{ .Data.issuing_ca }}
{{ end }}
EOF
        destination = "secrets/ca-cert.pem"
        change_mode = "restart"
      }

      env {
        TLS_CERT_FILE = "${NOMAD_SECRETS_DIR}/server-cert.pem"
        TLS_KEY_FILE  = "${NOMAD_SECRETS_DIR}/server-key.pem"
        TLS_CA_FILE   = "${NOMAD_SECRETS_DIR}/ca-cert.pem"
      }
    }
  }
}

2. Same runtime.exs config as the Kubernetes example — reads from environment variables.

Vault automatically renews certificates before they expire. The change_mode = "restart" triggers a graceful task restart when new certs are issued.

TLS Certificate Rotation

FerricStore reads certificate files at startup. When certificates are renewed:

• Kubernetes: cert-manager updates the Secret → pod restart picks up new certs
• Nomad: Vault template change_mode = "restart" triggers task restart
• Manual: Restart the FerricStore process after placing new certificates

FerricStore does not currently support hot-reloading TLS certificates without restart. Existing connections continue with the old certificate until they disconnect. New connections after restart use the new certificate.

Node-to-Node Security (Erlang Distribution)

What is Erlang distribution?

When FerricStore runs as a multi-node cluster, nodes talk to each other using Erlang distribution — a protocol built into the BEAM VM. This is not FerricStore-specific; it's how all Elixir/Erlang clusters communicate.

Everything between nodes goes over this channel:

• Raft consensus messages (AppendEntries, vote requests, commits)
• Cluster membership changes (node join/leave)
• GenServer.call across nodes
• Process monitoring (Process.monitor on remote processes)
• PubSub broadcasts

Why it's a security risk

Erlang distribution has two serious security problems out of the box:

1. No encryption. All traffic is plaintext. Anyone on the network can see your data, Raft log entries, and internal messages. This is especially dangerous across availability zones or any network you don't fully control.

2. The Erlang cookie gives full access. Nodes authenticate using a shared secret called the "cookie" (stored in ~/.erlang.cookie or set via --cookie). Any process that knows the cookie can connect to any node and execute arbitrary code — including System.cmd("rm", ["-rf", "/"]). The cookie is NOT a password; it's a symmetric key that grants root-level access to the BEAM VM.

  Without security:

  Attacker knows cookie → connects to node → :rpc.call(node, System, :cmd, ["rm", ["-rf", "/"]])
                                                 ↑
                                            full remote code execution

Step 1: Set a strong cookie

Never use the default cookie. Generate a random one:

# Generate a strong random cookie
openssl rand -base64 32 > /etc/ferricstore/cookie
chmod 400 /etc/ferricstore/cookie

In the FerricStore process environment:

export FERRICSTORE_COOKIE="$(cat /etc/ferricstore/cookie)"

Or in Kubernetes, store it as a Secret:

env:
  - name: FERRICSTORE_COOKIE
    valueFrom:
      secretKeyRef:
        name: ferricstore-secrets
        key: erlang-cookie

All nodes in the cluster must share the same cookie. They cannot communicate without it.

Step 2: Keep discovery scoped to the intended network

FerricStore disables libcluster discovery in the base config. Runtime discovery is enabled only when FERRICSTORE_NODE_NAME is set. If you use the default gossip discovery strategy, FerricStore binds gossip to 127.0.0.1 unless you explicitly set a different interface.

For production:

• Prefer FERRICSTORE_DISCOVERY=dns in Kubernetes or environments with stable service discovery.
• If you intentionally use gossip across hosts or containers, set FERRICSTORE_GOSSIP_IF_ADDR and FERRICSTORE_GOSSIP_MULTICAST_IF to the private cluster interface, not a public interface.
• Firewall the gossip UDP port, FERRICSTORE_GOSSIP_PORT (45892 by default), so only FerricStore nodes can send discovery packets.
• Always use a strong FERRICSTORE_COOKIE; gossip packets use that value as their discovery secret.

export FERRICSTORE_NODE_NAME=ferricstore@10.0.0.11
export FERRICSTORE_COOKIE="$(cat /etc/ferricstore/cookie)"
export FERRICSTORE_DISCOVERY=dns
export FERRICSTORE_DNS_NAME=ferricstore-headless

Step 3: Restrict distribution to a specific interface

By default, Erlang distribution listens on all interfaces (0.0.0.0). Bind it to a private interface:

# Only listen on the private network interface
export ELIXIR_ERL_OPTIONS="-kernel inet_dist_use_interface {10,0,0,1}"

Or restrict the port range (useful for firewall rules):

export ELIXIR_ERL_OPTIONS="-kernel inet_dist_listen_min 9100 -kernel inet_dist_listen_max 9110"

Then firewall those ports to only allow traffic from other FerricStore nodes.

Step 4: Enable TLS for distribution

For full encryption and mutual authentication, enable Erlang distribution over TLS. This encrypts all node-to-node traffic and requires each node to present a valid certificate.

1. Create an inet_tls.conf file:

%% inet_tls.conf
[{server, [
  {certfile, "/etc/ssl/ferricstore/node-cert.pem"},
  {keyfile, "/etc/ssl/ferricstore/node-key.pem"},
  {cacertfile, "/etc/ssl/ferricstore/ca-cert.pem"},
  {verify, verify_peer},
  {fail_if_no_peer_cert, true},
  {secure_renegotiate, true}
]},
{client, [
  {certfile, "/etc/ssl/ferricstore/node-cert.pem"},
  {keyfile, "/etc/ssl/ferricstore/node-key.pem"},
  {cacertfile, "/etc/ssl/ferricstore/ca-cert.pem"},
  {verify, verify_peer},
  {secure_renegotiate, true}
]}].

2. Start the BEAM with TLS distribution:

elixir --erl "-proto_dist inet_tls -ssl_dist_optfile /etc/ssl/ferricstore/inet_tls.conf" \
  -S mix run --no-halt

Or in a release (rel/env.sh.eex):

export ELIXIR_ERL_OPTIONS="-proto_dist inet_tls -ssl_dist_optfile /etc/ssl/ferricstore/inet_tls.conf"

3. Kubernetes example — same cert-manager certs can be used for both client TLS and distribution TLS. Mount to the same path and reference from both FerricStore config and inet_tls.conf.

4. Nomad example — Vault can issue separate node certificates:

template {
  data = <<EOF
{{ with secret "pki/issue/ferricstore-node" "common_name={{ env "attr.unique.hostname" }}" "ttl=720h" }}
{{ .Data.certificate }}
{{ end }}
EOF
  destination = "secrets/node-cert.pem"
  change_mode = "restart"
}

Important: All nodes in the cluster must use certificates signed by the same CA. A node presenting a certificate from a different CA will be rejected and unable to join the cluster.

See the Erlang/OTP documentation on SSL Distribution for the full reference.

Security checklist for multi-node deployments

Single-node deployments

If you're running FerricStore as a single node (the common case for embedded mode), none of the distribution security applies — there is no inter-node communication. Just configure client TLS and ACL.

Audit Logging

FerricStore maintains an audit trail of security-relevant events in an ETS ring buffer. Logged events include:

• ACL command denials (user, command, client IP, client ID)
• Authentication attempts (success and failure)
• ACL mutations (SETUSER, DELUSER)
• Configuration changes (CONFIG SET)

Viewing Audit Logs

Audit logs are accessible through the Ferricstore.AuditLog module in embedded mode:

Ferricstore.AuditLog.recent(20)

In standalone mode, audit events are also emitted as telemetry events that can be forwarded to external logging systems.

ACL Cache Invalidation

In standalone mode, each connection process caches the ACL user record for fast permission checks (~5ns vs ~250ns for ETS lookup on every command). When an ACL mutation occurs (SETUSER, DELUSER), a broadcast is sent via :pg to all connection processes, which invalidate their cached ACL records.

This means ACL changes take effect on the next command from each connection, with no need to disconnect and reconnect.

Security Best Practices

1. Set a password on the default user in production:

   ACL SETUSER default on >strong_random_password
   

2. Create dedicated users for each application with minimum required permissions:

   ACL SETUSER myapp on >apppassword +@read +@write -@dangerous ~myapp:*
   

3. Enable TLS for all production deployments:

   config :ferricstore, :require_tls, true

4. Use key patterns to restrict which keys each user can access:

   ACL SETUSER analytics on >pass +@read ~analytics:* ~metrics:*
   

5. Deny dangerous commands for non-admin users:

   ACL SETUSER appuser on >pass +@all -@dangerous -@admin ~app:*
   

6. Persist ACL so changes survive restarts:

   ACL SAVE
   

7. Monitor audit logs for unauthorized access attempts.

8. Use mutual TLS in zero-trust environments:

   config :ferricstore, :native_tls_ca_cert_file, "/etc/ssl/ca.pem"

Limitations

• No encryption at rest -- Bitcask data files are stored in plaintext on disk. Use filesystem-level encryption (LUKS, dm-crypt) for data-at-rest protection.
• ACL key pattern enforcement is still being finalized for all command types. String, hash, list, set, and sorted set commands enforce key patterns. Some administrative commands bypass key checks by design.
• No client certificate to ACL user mapping -- mutual TLS authenticates the client but does not automatically map to an ACL user. Use AUTH after connecting.