Going to production

A checklist for running Socketelx for real users, and the operating-system limits that most often catch people out. Read the Capacity planning guide for how much one server can carry.

Checklist

• [ ] Secrets. SOCKETELX_APP_SECRET is a long random value (openssl rand -hex 32).

  The built-in default is public. The server logs a warning if you leave it.

• [ ] TLS. Browsers need wss://. Terminate TLS at your proxy, load balancer or

  Cloudflare, or set `SOCKETELX_TLS_CERTFILE` and `SOCKETELX_TLS_KEYFILE`.

• [ ] Origins. Set SOCKETELX_ALLOWED_ORIGINS to your own sites.
• [ ] Open-file limit is at least your peak connections plus 1,000

  ([below](#open-file-limit-ulimit)).

• [ ] Proxy timeouts are longer than 60 seconds (below).
• [ ] /metrics is protected: set SOCKETELX_METRICS_TOKEN, or block the path at your proxy.
• [ ] Limits you want are set: SOCKETELX_MAX_CONNECTIONS, SOCKETELX_MAX_CONNECTIONS_PER_IP,

  `SOCKETELX_REST_RATE_LIMIT`. They are all off by default.

• [ ] Health check on GET / (the Docker image already has one).
• [ ] Graceful stop. Give the container at least 10 seconds to stop (Docker's default).

  On `SIGTERM` the server tells every client to reconnect, then exits.

• [ ] Two or more nodes if you cannot accept downtime during a restart or a crash

  ([Multi-server](multi-server.md)).

• [ ] Webhooks that matter: set SOCKETELX_WEBHOOK_QUEUE_PATH to a mounted volume so

  retries survive a restart, and make your receiver idempotent.

• [ ] You have load-tested your own setup, through your real proxy

  ([Load testing](load-testing.md)).

Open-file limit (ulimit)

Every WebSocket connection is one file descriptor. Operating systems often allow only 256 to 1,024 per process by default. When the limit is reached the server starts failing to accept connections (and Erlang may log emfile errors). It is the most common cause of "it works with 500 users and breaks at 1,000".

Set it comfortably above your peak connections (65,536 is a good start; use more if you plan for more than about 60,000 connections on one node).

Docker

docker run --ulimit nofile=65536:65536 -p 6001:6001 ghcr.io/socketelx/socketelx

Docker Compose

services:
  socketelx:
    image: ghcr.io/socketelx/socketelx
    ulimits:
      nofile:
        soft: 65536
        hard: 65536
    stop_grace_period: 15s

systemd

[Service]
LimitNOFILE=65536

Kubernetes has no per-pod ulimit setting. The limit comes from the container runtime on the node, and is usually already high. Check from inside the pod.

Check what the server actually got (do this, don't assume):

docker exec <container> sh -c 'cat /proc/1/limits | grep "open files"'

Your laptop (for local load tests): run ulimit -n 65536 in the terminal that starts the server and in the one that starts the load test. On macOS, if it is refused, run sudo launchctl limit maxfiles 65536 200000 first and open new terminals.

Operating-system network settings (Linux)

You rarely need these below about 20,000 connections per node. Above that, or behind a proxy, check them.

On macOS the equivalent of somaxconn is kern.ipc.somaxconn, and it defaults to 128. A burst of new connections on a laptop can be refused for that reason alone. Raise it with sudo sysctl -w kern.ipc.somaxconn=4096.

The proxy port limit

A proxy (nginx, a load balancer, cloudflared) opens one connection to Socketelx for every browser connection. All of them come from the proxy's one IP address to Socketelx's one IP and port, and a single source IP can use only about 28,000 ports by default (32768 to 60999). So one proxy in front of one Socketelx address tops out at about 28,000 connections, however big the machines are. Fixes: widen ip_local_port_range on the proxy host, give the proxy several upstream addresses, or run more nodes.

Reverse proxies and load balancers

• Pass the WebSocket upgrade headers and use HTTP/1.1. See deploy/nginx.conf.
• Keep the idle timeout longer than 60 seconds. Clients ping about every 30 seconds and the server closes a socket that stays silent for 60 seconds. We suggest 120 to 300 seconds. Cloud load balancers often default to 60 seconds, which is too tight.
• No sticky sessions are needed. A WebSocket stays on the node it connected to, and REST calls from your backend can go to any node.
• Behind a proxy every client looks like it has the proxy's address. If you use SOCKETELX_MAX_CONNECTIONS_PER_IP, also set SOCKETELX_CLIENT_IP_HEADER (cf-connecting-ip on Cloudflare, x-forwarded-for elsewhere) and make sure your proxy overwrites that header.
• Use a least-connections strategy rather than round robin: connections are long lived.

Memory

The server's memory grows with connections, and the Erlang VM keeps memory it has freed for reuse instead of handing it straight back to the OS. Set a container memory limit of at least twice your expected steady-state use, or the container may be killed under a burst. Watch socketelx_connections next to the process's memory.

Rolling deploys and restarts

• Restart nodes one at a time and wait for socketelx_connections on the restarted node to recover before the next one. A node that stops tells its clients to reconnect, and they land on the others.
• Clients reconnect on their own (pusher-js backs off and retries). A large node restarting means many clients reconnecting together, which is the reason to keep somaxconn generous and to restart one node at a time.
• Nothing needs backing up: the server keeps no data on disk, apart from the optional webhook queue.

Monitoring

Turn on SOCKETELX_METRICS=true and scrape /metrics from every node. Useful alerts:

Redis or Erlang nodes?

For several servers, Redis is the better-tested path and keeps nodes stateless. The Erlang cluster mode needs no extra service but is experimental. See Multi-server. Test both on your own network; the load-testing guide shows how to run the same test against two nodes.

Troubleshooting capacity problems