Hybrid Logical Clock (HLC) for FerricStore.

An HLC combines a physical wall-clock component (milliseconds since epoch) with a logical counter to produce timestamps that are:

1. Monotonically increasing -- even when the wall clock is not (NTP corrections, VM migration, etc.).
2. Causally ordered -- merging a remote timestamp via update/1 ensures happens-before relationships are preserved across nodes.
3. Close to real time -- the physical component tracks System.os_time(:millisecond) and only diverges when the wall clock jumps backward or a remote node is ahead.

Spec reference (2G.6)

• HLC is piggybacked on Raft heartbeats.
• Inter-node TTL precision is bounded to Raft heartbeat RTT (~10 ms).
• A telemetry warning is emitted at 500 ms drift between HLC physical and wall clock.
• TTL-sensitive reads are rejected at 1 000 ms drift.
• The HLC millisecond component is used for Stream ID generation (XADD *).
• HLC timestamps are stamped on commands before they enter Raft -- they are not computed inside apply().

Architecture

The hot-path functions now/0 and now_ms/0 are lock-free: they use a single :atomics slot (stored in :persistent_term) instead of a GenServer.call. Physical ms and logical counter are packed into one 64-bit integer, eliminating the two-slot race that broke monotonicity under contention.

The GenServer is retained only for:

• update/1 -- merging remote timestamps from Raft heartbeats (~6 calls/sec). This must be serialized to avoid lost-update races between concurrent merges.
• Process supervision -- the init/1 callback creates the atomics ref and stores it in :persistent_term.

Packed layout in a single unsigned 64-bit atomic:

|-- physical_ms (48 bits) --|-- logical (16 bits) --|

48 bits covers ~8,920 years of milliseconds. 16 bits allows 65,535 increments per millisecond. If logical overflows, it spills into the physical bits — effectively advancing the clock by 1 ms, which is safe.

Usage

The application supervision tree starts a single named HLC process (Ferricstore.HLC). Other modules obtain timestamps via:

{physical_ms, logical} = Ferricstore.HLC.now()
ms = Ferricstore.HLC.now_ms()

When receiving a Raft heartbeat with a piggybacked timestamp:

:ok = Ferricstore.HLC.update(remote_timestamp)

Timestamp representation

A timestamp is a 2-tuple {physical_ms, logical} where:

• physical_ms -- milliseconds since Unix epoch (same scale as System.os_time(:millisecond))
• logical -- a non-negative integer counter that disambiguates events within the same physical millisecond

Timestamps are ordered lexicographically: physical first, then logical.

Cross-node synchronization

HLC sync across nodes currently happens via Raft heartbeats (~6/sec), giving ~150ms max drift. This is sufficient for all current use cases:

• Read-your-writes on the writing node: guaranteed by ra:process_command(reply_from: :local).
• Cross-node reads after quorum write: guaranteed by Raft consensus — all quorum nodes applied the command before the write returns.
• TTL expiry: second-granularity TTLs are unaffected by 150ms drift.
• Cross-shard WATCH: uses value hashing, not timestamps.

Per-command HLC stamping (not wired up)

The state machine can handle commands wrapped as {inner_command, %{hlc_ts: {physical_ms, logical}}}. When apply/3 processes a wrapped command, it calls update/1 to merge the leader's clock into the follower's HLC. This gives per-command causal sync (~0ms drift) instead of heartbeat-level sync (~150ms drift).

Nothing sends wrapped commands today. Enable if any of these arise:

1. Sub-second TTLs with cross-node reads — a key with PX 100 (100ms TTL) could appear alive on one node and expired on another within the 150ms drift window.
2. Cross-node causal ordering — if a feature needs "write A happened-before write B" guarantees across nodes beyond Raft log ordering (e.g., conflict resolution in multi-leader setups).
3. Observed drift exceeding 500ms — if heartbeat intervals increase due to network issues or overloaded nodes.

To enable, stamp commands in Router.raft_write/4 before submitting:

stamped = {command, %{hlc_ts: HLC.now()}}
:ra.process_command(shard_id, stamped, opts)

Cost: ~50ns per write (now/0 on leader) + one update/1 GenServer call per follower during apply/3. CockroachDB and YugabyteDB stamp every RPC and consider the cost negligible. Redis and Kvrocks do not have cross-node transactions or HLC.

Graceful fallback

When the HLC GenServer has not been started (e.g. in unit tests that exercise command modules without the full application), now/0 and now_ms/0 fall back to System.os_time(:millisecond) with a logical counter of 0.