LeanLmdb

A lean, binary-first Elixir wrapper for LMDB, built with Rustler 0.38, RustlerPrecompiled 0.8, and heed 0.22.1. It is intended for bounded, rebuildable local derived stores—not as a source-of-truth event store.

Installation

Add the Hex package to mix.exs:

def deps do
  [{:lean_lmdb, "~> 0.1.0"}]
end

Installation downloads the matching NIF from the v0.1.0 GitHub release and verifies its SHA-256 digest against the packaged checksum-Elixir.LeanLmdb.Native.exs manifest. The package retains locked Rust source as an explicit fallback for unsupported targets and local source builds.

NIF 2.16 artifacts are built and raw-smoked for this exact matrix:

Set LEAN_LMDB_BUILD=1 (true, yes, and on are also accepted, case-insensitively) at compile time to force the packaged-source fallback. MIX_ENV=test always forces source and enables test-support NIFs. For global QA across all RustlerPrecompiled dependencies, set RUSTLER_PRECOMPILED_FORCE_BUILD_ALL=true. Consumers forcing fallback must also make the package's optional build-only dependency available, for example {:rustler, "~> 0.38.0", runtime: false}. Source builds require Elixir 1.20, Erlang/OTP 29, Cargo/Rust 1.91, and the platform C compiler/linker. Linux musl builds dynamically link the musl CRT and are validated in matching Alpine Elixir containers. Runtime environments must use a local filesystem.

Quickstart

All names, keys, values, CAS payloads, and continuation tokens are binaries; there is no Elixir-term serialization.

path =
  Path.join(
    System.tmp_dir!(),
    "lean_lmdb_example_#{System.unique_integer([:positive, :monotonic])}"
  )
{:ok, env} = LeanLmdb.open(path, map_size: 64 * 1024 * 1024)
{:ok, users} = LeanLmdb.create_database(env, "users")
{:ok, cache} = LeanLmdb.create_database(env, "cache")

:not_found = LeanLmdb.get(users, "user:1")
:ok = LeanLmdb.put(users, "user:1", <<0, 255>>)
{:ok, <<0, 255>>} = LeanLmdb.get(users, "user:1")
:ok = LeanLmdb.delete(users, "user:1")

:ok = LeanLmdb.write_batch(env, [
  {:put, users, "user:1", "Ada"},
  {:put, users, "user:2", "Grace"},
  {:delete, cache, "stale"}
])

:ok = LeanLmdb.compare_exchange(users, "user:3", :missing, {:put, "Lin"})
{:conflict, "Lin"} =
  LeanLmdb.compare_exchange(users, "user:3", :missing, {:put, "Other"})

{:ok, first, token} = LeanLmdb.scan(users, prefix: "user:", limit: 2)
{:ok, second, :done} =
  LeanLmdb.scan(users, prefix: "user:", limit: 2, continuation: token)
true = first ++ second == [{"user:1", "Ada"}, {"user:2", "Grace"}, {"user:3", "Lin"}]

Use a unique path in real tests and remove it only after all handles are unreachable. There is no force-close API.

Public result contracts

See the generated API documentation for each function's validation rules and stable reasons. Native failures return atoms, never platform error prose.

Lifecycle and transaction containment

Within one BEAM OS process, compatible opens of the same canonical path share native environment state. Opaque Environment handles and immutable Database handles may remain live across calls; a database handle retains the shared environment state. No heed transaction or cursor is an Elixir resource. Every transaction/cursor starts and ends in one dirty-I/O NIF call.

Reads copy values into BEAM-owned binaries before ending the transaction. Scans also copy every key, value, and token. LeanLmdb is therefore intentionally not an end-to-end zero-copy API. NIF hot upgrade while native handles exist is not supported; restart the BEAM when replacing the native library.

Environment defaults, map sizing, and durability

open/2 defaults to a 64 MiB map, 16 named databases, 126 readers, writable access, and create: true. map_size is fixed, page-size aligned, and limited to 1 MiB..1 TiB. max_dbs is 1..1024 and max_readers is 1..4096. LeanLmdb does not resize maps, retry failed operations, or expose relaxed LMDB flags. Size the map and reader/database limits for the whole workload before opening.

Writes use heed/LMDB's default synchronized durability flags: no NO_SYNC, NO_META_SYNC, WRITE_MAP, or NO_LOCK mode is exposed. A successful write means LMDB committed it under those defaults; storage hardware, OS, and filesystem guarantees still apply. There is no separate flush API. Common capacity/operational results include :map_full, :map_resized, :readers_full, :transaction_full, :out_of_memory, and :io_error.

CRUD, atomic mutation, and single-writer behavior

Keys are non-empty and at most LMDB's runtime maximum (currently bounded by 511 bytes here). Values may be empty and can contain arbitrary bytes. put/3 overwrites; delete/2 is idempotent. A failed write aborts its transaction.

Batches contain at most 10,000 operations and 64 MiB of aggregate key/value input. All database handles must belong to the supplied environment. The full batch is validated before one write transaction, applied in list order, and committed once. CAS distinguishes :missing from <<>> and returns the exact copied current value on conflict.

LMDB allows concurrent readers but exactly one active writer per environment, including across OS processes. A dirty-I/O call can wait for that writer; short batches improve useful work per serialized commit without changing durability.

Bounded scans and tokens

scan/2 is forward-only in raw binary-key order. Choose prefix: binary or a range using from, to, from_inclusive (default true), and to_inclusive (default false). Do not mix prefix and range options. limit defaults to 100 and is 1..10,000. max_bytes defaults to 1 MiB and is 1..64 MiB, counting key plus value bytes. If the first eligible row is larger than the selected budget, the call returns {:error, :row_too_large}; page output never exceeds the byte or row bound. Because 64 MiB is the maximum budget, a key/value row larger than 64 MiB cannot be traversed by scan/2; use smaller/chunked values or a different store design.

A token is an opaque, versioned, process-local binary with a keyed integrity tag, not a cryptographic credential. It contains no pointer or cursor. It binds to the native environment resource ID, database name, and original bounds; limit and max_bytes may change. Modified/mismatched tokens return :invalid_continuation; tokens from a released environment return :stale_continuation. Tokens do not survive a BEAM restart.

Each page has a fresh read transaction, so pages do not share one snapshot. Resume is exclusive after the last emitted key. Writes between pages can add, change, or remove later rows; insertions at or before the last emitted key will not appear. An unchanged database traverses exactly once without duplicates.

Multi-process operation and stale readers

Each BEAM OS process has its own registry and heed handle. LMDB coordinates via its data and lock files. Configure all processes with the same map/database/ reader limits; first-opener and process-local details otherwise can produce :map_resized or incompatible behavior. A commit becomes visible to another process when that process starts a later transaction.

An abrupt writer death cannot publish a partial LMDB transaction, but recovery is not corruption repair. A killed reader can leave a stale lock-table slot, causing file growth or eventually :readers_full. clear_stale_readers/1 uses LMDB's reader check to remove dead-process slots only and is safe to repeat. Never delete the lock file while any process has the environment open.

The deterministic integration suite uses independent BEAM processes, explicit file barriers, bounded polling, and Unix hard kills; it does not infer recovery from sleeps.

Filesystem and operational caveats

Use only a stable local filesystem. NFS, SMB, distributed/network mounts, and moving, replacing, truncating, or externally editing open environment files are unsupported. Available disk and virtual address space and LMDB/platform limits still apply. Keep backups and the authoritative event/source data separately; LeanLmdb has no backup, replication, corruption repair, distributed consensus, automatic resize, or native hot-upgrade protocol.

The API fits a derived store when data is binary, locally rebuildable, bounded scans are sufficient, and serialized short writes are acceptable. It is a poor fit when the database is authoritative, values need streaming/chunking, remote storage is required, or sustained parallel write transactions are required.

Source development

Install the versions above, fetch both dependency sets once, then run the single quality gate:

mix deps.get
cargo +1.91.0 fetch --manifest-path native/lean_lmdb_nif/Cargo.toml --locked
bin/qa_check.sh

After dependencies exist, the gate globally forces source fallback, forces Hex and Cargo offline, builds docs with warnings as errors, runs Elixir tests twice, runs all-feature Rust lint and tests, and performs a release build without test-support. It also checks an optional checksum manifest is packaged unchanged, compiles the unpacked package from its included Rust source, then runs its package-built NIF through native_version/0 and isolated CRUD. Every Mix/Cargo command has a portable per-command deadline (900 seconds by default, configurable with QA_COMMAND_TIMEOUT_SECONDS).

Maintainer release flow

Precompiled publication is explicit and draft-first. The workflow never runs on a tag push, so it cannot recurse. For a version already set in mix.exs and the native Cargo.toml:

git push origin main
gh workflow run precompiled-release.yml --ref main -f publish=true
gh run list --workflow precompiled-release.yml --limit 1
gh run watch RUN_ID --exit-status

gh release view v0.1.0 --json tagName,targetCommitish,isDraft,isPrerelease,assets
mix rustler_precompiled.download LeanLmdb.Native --all --print

The matrix uses target-aware Cargo caches. Every archive is directly loaded on a matching runtime before aggregation; Linux musl uses Alpine. The aggregator accepts exactly seven expected archives, creates a draft from the dispatched commit, verifies GitHub's SHA-256 asset digests, then publishes. A failure after draft creation removes only the draft and tag created by that run.

The generated checksum-Elixir.LeanLmdb.Native.exs is mandatory for the normal precompiled path. Review and commit it, rerun bin/qa_check.sh, then push and wait for the CI precompiled-consumer matrix. Those jobs compile the unpacked Hex package from clean downstream projects with cargo and rustc replaced by failing shims. Use gh run cancel RUN_ID immediately for an obsolete or known- bad run, inspect failures with gh run view RUN_ID --log-failed, and retry only after committing the fix.