Erlang distribution over Iroh

View Source

IrohBeam can act as an opt-in OTP 29 distribution carrier. Iroh supplies an encrypted, endpoint-key-authenticated QUIC byte stream; OTP still supplies the normal cookie handshake, distribution encoding, RPC, links, monitors, ticks, and node lifecycle.

The carrier is not membership. It never chooses peers or calls Node.connect/1. Every permitted peer is configured exactly, and application code decides which links to establish.

Security layers

Keep these separate operationally:

  1. Each VM owns a distinct persistent Iroh identity. Its public endpoint ID is safe to share; its private key is not.
  2. An ID, address, or endpoint ticket supplies reachability. A relay token only grants access to that relay.
  3. The static peer map binds one existing Erlang node atom to one authenticated endpoint ID. Unknown IDs and mismatched name/key claims are rejected before OTP receives peer-controlled node-name text.
  4. Erlang cookies remain mandatory and must match.
  5. Membership code explicitly calls Node.connect/1 and handles reconnects.

Dynamic startup

Launch the unnamed VM with the carrier selected and EPMD disabled:

elixir --erl "-proto_dist iroh -no_epmd" -S mix run --no-halt

After runtime configuration is available:

peer_id = IrohBeam.EndpointId.parse!(System.fetch_env!("PEER_ID"))

{:ok, _net_kernel} =
  IrohBeam.Distribution.start(
    name: :"api@east",
    name_domain: :shortnames,
    identity: {:file, "data/api.iroh"},
    network: :n0,
    peers: %{:"worker@west" => peer_id}
  )

Node.set_cookie(:replace_with_a_deployment_secret)
true = Node.connect(:"worker@west")
:pong = Node.ping(:"worker@west")

-proto_dist iroh must be present before boot even for dynamic startup, because OTP reads protocol selection from init arguments. -no_epmd is required; the carrier never starts or registers with EPMD.

Distribution.stop/0 is supported for dynamically started distribution. Peer configuration is immutable while running; stop fully before installing a new map.

Early named startup

A release can start named distribution during kernel boot:

erl -pa /path/to/iroh_beam/ebin \
  -proto_dist iroh -no_epmd -sname api@east ...

The same keyword configuration must already exist as boot-time application environment under :iroh_beam, :distribution. A generated sys.config works. A configuration provider or runtime.exs evaluated after kernel distribution starts is too late. Early command-line distribution cannot be stopped through Distribution.stop/0 when OTP rejects that operation.

Static targets

Typed IDs, addresses, and tickets are accepted, as are explicit serialized forms:

peers: %{
  :"one@west" => {:id, endpoint_id_text},
  :"two@north" => {:ticket, endpoint_ticket_text},
  :"three@south" =>
    {:addr,
     %{
       endpoint_id: endpoint_id_text,
       relay_urls: ["https://relay.example/"],
       ip_addrs: []
     }}
}

Aliases are exact. Wildcards, allow-all admission, dynamic mutation, and peer-supplied atom creation are not supported.

Private relay-only distribution

{:ok, relay} =
  IrohBeam.Relay.new("https://relay.example/", token: System.fetch_env!("RELAY_TOKEN"))

{:ok, _} =
  IrohBeam.Distribution.start(
    name: :"api@east",
    identity: {:file, "data/api.iroh"},
    network: {:custom, [relay]},
    direct_ip: false,
    peers: %{
      :"worker@west" =>
        {:addr,
         %{
           endpoint_id: System.fetch_env!("WORKER_ID"),
           relay_urls: ["https://relay.example/"],
           ip_addrs: []
         }}
    }
  )

direct_ip: false removes IP transports, rather than preferring the relay. peer_info/1 reports the authenticated endpoint and selected :relay path. Relay-token failure occurs before Iroh connectivity; endpoint admission occurs before the OTP handshake; cookie failure occurs in OTP.

Limits and lifecycle

The data plane keeps one bounded read and one flow-controlled write in flight per link. Defaults are a 64 KiB receive chunk and a 16 MiB maximum distribution frame. Packet lengths are checked before body retention. OTP's negotiated fragmentation normally keeps frames much smaller than the maximum.

Ticks use OTP's net_ticktime and net_tickintensity. A relay or peer outage can produce nodedown; IrohBeam does not silently reconnect or heal topology. After reachability returns, application code explicitly calls Node.connect/1 again. status/0 and peer_info/1 expose safe lifecycle, path, byte, and frame counters without cookies, keys, tokens, tickets, or packet contents.

Testing boundaries

The repository's relay suite starts three separate OS processes with distinct keys, disables direct IP, forms an explicit triangle, verifies relay paths, runs RPC/links/monitors/large traffic/ticks, stops the relay, proves new links fail, then restarts peers and reconnects. This is a real separate-BEAM and relay-only proof, but it does not simulate every NAT, firewall, latency profile, physical network, or production relay topology.