Bidirectional mapping from values to unique integer IDs.
Makes equality comparison O(1) for interned values, which directly impacts early cutoff performance. Every string, identifier, and type that flows through the query graph should be interned as early as possible.
Interned values use integer IDs, never atoms, to avoid BEAM atom table exhaustion (see D10).
Concurrency
All operations are thread-safe. Concurrent calls to intern/2 with the
same value are resolved lock-free via :atomics.add_get/3 for ID
allocation and :ets.insert_new/2 as a compare-and-swap. IDs need not
be contiguous — the counter may advance past IDs that lost the CAS race.
Restored tables load on first use
A table restored from an encoded snapshot (encode_snapshot/1) keeps
its rows encoded until an operation misses: the first intern/2,
lookup/2 or resolve/2 that does not find its answer loads them, then
looks again. A warm run of a large scry project restores 170,000
interned symbols and reads none of them; loading both tables eagerly was
a quarter of restoring its manifest. See restore/2 for why a miss is
the right trigger, and why concurrent loads are safe.
Summary
Types
A persisted intern table with its forward rows in the external term
format, tagged with its own version. restore/2 keeps the rows encoded
until the table is first used.
A persisted intern table: the forward rows and the ID counter, tagged with the snapshot format's version.
Functions
Deletes both ETS tables. Called during database shutdown.
Captures the forward table, encoded, and the counter: the snapshot a manifest persists.
Interns a value, returning its integer ID.
Checks if a value is already interned without interning it.
Creates a new intern table pair.
Resolves an integer ID back to its original value.
Resolves an integer ID back to its original value.
Restores a table from a snapshot: one produced by snapshot/1, or an
encoded one produced by encode_snapshot/1.
Returns the number of interned values.
Captures the forward table and the counter for manifest persistence.
Types
@type encoded_snapshot() :: %{ version: 3, forward: binary(), counter: non_neg_integer() }
A persisted intern table with its forward rows in the external term
format, tagged with its own version. restore/2 keeps the rows encoded
until the table is first used.
@type id() :: pos_integer()
@type snapshot() :: %{ version: 2, forward: [{term(), id()}], counter: non_neg_integer() }
A persisted intern table: the forward rows and the ID counter, tagged with the snapshot format's version.
Only one direction is stored. Every value appears in both tables, so
persisting both stored each value twice — on a large scry project the
interned symbols were a third of the manifest. The forward table is the
one kept because it is authoritative: an ID only becomes live when its
{value, id} row wins :ets.insert_new/2, while the reverse table can
briefly hold the orphaned ID of a process that lost that race. The
reverse table is rebuilt from it on restore.
@type t() :: %Roux.Intern{ counter: :atomics.atomics_ref(), forward: :ets.tid(), reverse: :ets.tid() }
Functions
@spec destroy(t()) :: :ok
Deletes both ETS tables. Called during database shutdown.
@spec encode_snapshot(t()) :: encoded_snapshot()
Captures the forward table, encoded, and the counter: the snapshot a manifest persists.
A table restored from an encoded snapshot that nothing has been interned into since hands back the encoding it was restored from, whether or not it has been read: its rows are exactly the restored ones. Once a value is interned, the restored encoding is dropped and the table is encoded as it stands.
Interns a value, returning its integer ID.
If the value is already interned, returns the existing ID. Thread-safe: concurrent calls with the same value return the same ID.
Checks if a value is already interned without interning it.
Returns {:ok, id} if the value is interned, :error otherwise.
Creates a new intern table pair.
The name argument is for debugging and ETS introspection only — tables
are unnamed to avoid atom exhaustion.
Resolves an integer ID back to its original value.
Returns {:ok, value} if the ID exists, :error otherwise.
Resolves an integer ID back to its original value.
Raises Roux.Intern.UnknownIdError if the ID has not been interned.
@spec restore(t(), snapshot() | encoded_snapshot()) :: :ok
Restores a table from a snapshot: one produced by snapshot/1, or an
encoded one produced by encode_snapshot/1.
A snapshot/1 snapshot fills both tables now, rebuilding the reverse
table from the forward rows. An encoded snapshot sets the counter now
and leaves the rows encoded, pending, until an operation misses (see
"Restored tables load on first use"). A miss is the right trigger
because nothing can be found before the rows are loaded, and nothing
can be interned anew before a miss has loaded them: after a miss, an
operation makes sure no rows are pending (loading them if they are)
and only then looks again, and that second answer is final. A new
value therefore takes its ID after every restored row is in place, so
it never duplicates a restored value. Processes that miss at the same
time each load the rows; the rows they insert are identical, and no
restored row is ever rewritten afterwards (IDs past the restored
counter belong to new values), so a second load changes nothing.
Until the rows are loaded they sit in the reverse table under ID 0,
which no interned value has (IDs start at 1), and the encoding stays
there after (see encode_snapshot/1). Read a restored table through
this module: its ETS tables are empty until it is used.
Used during manifest loading. The caller must ensure the tables are empty
or freshly created. Raises ArgumentError for a snapshot in any other
format (such as the unversioned format that stored both tables).
@spec size(t()) :: non_neg_integer()
Returns the number of interned values.
Captures the forward table and the counter for manifest persistence.
The snapshot is versioned; restore/2 refuses any other format rather
than misreading it.