Performance & Scaling

MqttX is architected to scale from tens of thousands to roughly a million concurrent device connections on a single BEAM node, depending on hardware and workload — see Capacity Planning for what each instance size actually supports and where the ceiling comes from. This guide explains the architectural decisions and optimizations that make this possible.

Architecture Overview

Each MQTT connection is a lightweight Erlang process (~2KB initial heap, ~20KB total with connection state and socket overhead). The BEAM VM's preemptive scheduler distributes these processes across all available CPU cores. At 100k connections, connection state alone is roughly 2GB; size the machine above that — see Capacity Planning for the headroom a real deployment needs.

The key bottlenecks at scale are not the number of connections, but the hot paths that execute on every message:

Topic Router

The router uses a trie (prefix tree) keyed by topic segments. Given a subscription to sensors/+/temperature, the trie looks like:

root
└── "sensors"
    └── :single_level  (+)
        └── "temperature"
            └── subscribers: %{client1 => %{qos: 1}}

Matching walks the trie, branching into up to 3 children at each level: exact segment match, single-level wildcard (+), and multi-level wildcard (#). This is O(L + K) where L is the topic depth and K is the total matching subscribers — independent of total subscription count.

Impact at scale:

The trie also stores a by_client index mapping each client to its subscriptions, making unsubscribe_all (client disconnect cleanup) efficient without scanning the entire subscription list.

Packet Encoding

All socket sends use Codec.encode_iodata/2 which returns an iolist — a nested list of binaries that :gen_tcp.send/2 and :ssl.send/2 accept natively. This avoids a final IO.iodata_to_binary/1 copy.

For a typical 50-byte PUBLISH packet, this saves one 50-byte allocation and copy per send. At 100k messages/second, that's 5MB/s of avoided garbage collection pressure.

Codec benchmarks (Apple M4 Pro):

Buffer Handling

TCP delivers data in arbitrarily-sized chunks. In the common case, a complete MQTT packet arrives in a single TCP frame and the receive buffer is empty. The optimized path:

buffer = case state.buffer do
  <<>> -> data          # Common case: no copy, just use the new data
  buf  -> buf <> data   # Partial packet pending: concat
end

The <<>> match is a constant-time check. When the buffer is empty (the majority case with typical MQTT packet sizes < TCP MSS), we skip binary concatenation entirely. The rest returned by Codec.decode is already a zero-copy sub-binary reference into the original data.

Callback Dispatch

Elixir's function_exported?/3 performs a module lookup on each call. For optional callbacks like handle_info/2, handle_puback/2, and handle_mqtt_event/3, this check runs on every incoming packet. MqttX computes these once at connection init:

# Computed once in handle_connection/init:
handler_has_handle_info: function_exported?(handler, :handle_info, 2),
handler_has_handle_puback: function_exported?(handler, :handle_puback, 2)

# Then used as a simple boolean check per packet:
if state.handler_has_handle_puback do
  # ...
end

Flow Control

MQTT 5.0's receive_maximum limits how many unacknowledged QoS 1/2 messages can be in flight simultaneously. Both client and server enforce this with a direct counter:

# Server-side: check before accepting incoming QoS 2 PUBLISH
if state.inflight_count >= state.server_receive_maximum do
  # Send PUBREC with reason code 0x93 (Receive Maximum exceeded)
end

The counter is incremented when sending PUBREC (QoS 2 received) and decremented when sending PUBCOMP (QoS 2 complete) or when entries are dropped after max retries.

Retained Message Delivery

When a client subscribes, the server delivers matching retained messages from ETS. The optimized approach:

1. Exact topic subscriptions (no wildcards): Direct ets.lookup/2 — O(1) per subscription.
2. Wildcard subscriptions: Table scan with pre-normalized topic lists. Topic filters are normalized once before the scan, and retained messages store a pre-computed normalized list alongside the string key, avoiding String.split/2 in the inner loop.

For a server with 10,000 retained messages and a client subscribing to 5 exact topics, this reduces from 50,000 comparisons (5 filters x 10,000 messages) to 5 hash lookups.

Capacity Planning

Read this first. The numbers below are derived from architectural analysis and the codec benchmarks above — not from end-to-end load tests. This project does not yet ship a load harness, so treat them as a starting point for your own measurements, not as guarantees. Real capacity depends on message sizes, subscription fan-out, retained-store size, TLS vs plaintext, and above all the work your handler callbacks do.

Per-device resource usage

Budget roughly 20–25 KB of system RAM per connected device, split between the BEAM and the kernel:

The kernel share is the one most often forgotten: it does not appear in the BEAM's memory reporting, but it comes out of the same machine's RAM. At 500K connections it alone is 2–4 GB.

Sizing method

Do not allocate 100% of RAM to connection state. A workable rule:

devices ≈ (total_RAM × 0.60) / 22 KB

The 40% reserve covers the BEAM runtime itself, ETS tables (the retained-message store — up to :max_retained_messages, 100K topics by default — plus the subscription trie), the binary heap for in-flight payloads, OS page cache, and headroom for reconnect storms, which transiently allocate far above steady state. Machines sized to their steady-state ceiling fall over during exactly the event you most need them to survive: a mass reconnect after a network blip.

Instance sizing — idle-ish IoT (~1 msg/min)

The right-hand columns diverge on purpose. Below ~500K connections RAM is the limit and the arithmetic holds. Above it the bottleneck moves somewhere else entirely, and buying more RAM stops helping:

• Kernel socket memory grows with connections, not cores: 1M sockets is 4–8 GB of kernel buffers on top of BEAM RSS.
• Shared ETS tables — the retained store and the subscription router are single tables every connection process reads. Read concurrency is enabled, but write-heavy retained or subscribe/unsubscribe churn contends regardless of how many cores you add.
• Accept and handshake rate, not steady state, is what fails first on a large node. A mass reconnect of 1M devices means 1M CONNECTs (and TLS handshakes, which are CPU-expensive) arriving in seconds.
• Failure domain. One node holding 1M devices is one restart away from a 1M-device reconnect storm against itself.

For these reasons, past roughly 500K connections per node, horizontal scaling is usually the better engineering answer than a larger instance — several 8-vCPU nodes behind a load balancer beat one 16-vCPU node, and give you somewhere to fail over to. See Beyond a single node.

Every row above ~65,000 connections requires raising the BEAM port limit (+Q) and file-descriptor limits; rows above 262,144 also require +P. See VM Tuning and OS Tuning — these are not optional.

Message-rate ceilings

When devices are chatty, CPU binds before RAM. Per-vCPU throughput, assuming small (≈50-byte) QoS 0 payloads and a handler that does negligible work:

Scaling across cores is sub-linear — assume roughly 80–85% efficiency per doubling beyond 4 vCPUs, due to scheduler rebalancing, per-process GC pauses, and ETS contention:

These assume plaintext TCP. TLS adds per-handshake CPU cost (significant during reconnect storms) and a smaller per-message cost; budget conservatively if devices reconnect frequently on flaky cellular links.

System-level constraints

At high connection counts, OS and kernel limits usually bind before BEAM ones:

• File descriptors: one per connection. Raise ulimit -n above your target (see OS Tuning).
• BEAM port limit: every socket is a port, and the default is only 65,536 — the first hard wall you hit. Raise with +Q. The default process limit (262,144) binds later; raise with +P.
• Kernel socket buffer memory: ~4–8 KB per socket by default, outside BEAM RSS. At 500K connections that is 2–4 GB; at 1M, 4–8 GB. Tune net.ipv4.tcp_rmem and tcp_wmem downward for many-idle-connection workloads.
• Client-side port exhaustion: a client host can open ~64K outbound connections per destination IP:port pair. This does not cap a broker — a listening server is limited to ~64K connections per distinct client IP, since each connection is identified by the full 4-tuple. It matters when load-testing from a small number of source hosts, where you will hit the limit on the generator long before the broker.
• Accept backlog: raise net.core.somaxconn so reconnect bursts are not dropped at the listen queue.

Beyond a single node

Past ~500K connections, consider clustering multiple BEAM nodes behind a load balancer. The constraints at this scale are fault isolation (a single node crash affects all connected devices) and system-level limits described above. A multi-node setup with 3–5 nodes provides both capacity and redundancy.

Deployment Guidelines

Single Node

See Capacity Planning above for per-instance figures and the sizing method behind them. Note that QoS 2 carries higher overhead than the numbers there imply, because of its four-step handshake.

VM Tuning

For high connection counts, tune the BEAM scheduler:

# Use all available cores
elixir --erl "+S $(nproc)" -S mix run

# Increase process limit (default 262144)
elixir --erl "+P 1000000" -S mix run

# Increase port limit for socket handles
elixir --erl "+Q 200000" -S mix run

Or in rel/vm.args:

+S 8:8
+P 1000000
+Q 200000
+stbt db
+sbwt very_long

OS Tuning

# Increase file descriptor limit (each connection = 1 fd)
ulimit -n 200000

# Linux: increase socket buffer sizes
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
sysctl -w net.ipv4.tcp_wmem="4096 87380 16777216"

# Increase ephemeral port range
sysctl -w net.ipv4.ip_local_port_range="1024 65535"

Rate Limiting

For production deployments, enable rate limiting to protect against misbehaving clients and connection storms:

MqttX.Server.start_link(MyApp.MqttHandler, [],
  transport: MqttX.Transport.ThousandIsland,
  port: 1883,
  rate_limit: [
    max_connections: 100,    # per second
    max_messages: 1000       # per client per second
  ]
)

The rate limiter uses ETS with atomic update_counter for lock-free concurrent access. Counters reset automatically each interval window.

Transport Selection

Both ThousandIsland and Ranch are battle-tested for high connection counts:

Monitoring

Use the telemetry events (see Telemetry guide) to track:

• Connection rate: [:mqttx, :server, :client_connect, :stop] counter
• Message throughput: [:mqttx, :server, :publish] counter
• Publish latency: [:mqttx, :client, :publish, :stop] duration histogram
• Payload sizes: [:mqttx, :server, :publish] payload_size distribution
• Connection errors: [:mqttx, :client, :connect, :exception] counter by reason

← Back to the documentation index