trove
An embedded, crash-safe key-value store for Gleam.
trove stores data in an append-only B+ tree on disk. Every write appends new nodes and creates a new root. Old data is never overwritten, which gives you crash safety, zero-cost MVCC snapshots, and single-writer / multiple-reader concurrency backed by an OTP actor.
Quick Start
import gleam/string
import trove
import trove/codec
let config = trove.Config(
path: "./my_db",
key_codec: codec.string(),
value_codec: codec.string(),
key_compare: string.compare,
auto_compact: trove.AutoCompact(min_dirt: 1000, min_dirt_factor: 0.25),
auto_file_sync: trove.AutoSync,
call_timeout: 5000,
)
let assert Ok(db) = trove.open(config)
trove.put(db, key: "language", value: "gleam")
let assert Ok("gleam") = trove.get(db, key: "language")
trove.close(db)
Types
Controls automatic compaction behavior. When auto-compaction is enabled,
compaction triggers after a write if both min_dirt and min_dirt_factor
thresholds are exceeded simultaneously.
The recommended default is AutoCompact(min_dirt: 1000, min_dirt_factor: 0.25), which keeps disk usage bounded without compacting too aggressively.
Note: Auto-compaction runs synchronously inside the database actor.
While compaction is in progress, all other operations (reads, writes,
snapshots) are queued and may time out on large databases. For
latency-sensitive workloads, prefer NoAutoCompact and call compact
manually from a separate process with an appropriate timeout.
pub type AutoCompact {
AutoCompact(min_dirt: Int, min_dirt_factor: Float)
NoAutoCompact
}
Constructors
-
AutoCompact(min_dirt: Int, min_dirt_factor: Float)Enable auto-compaction.
min_dirtis the minimum number of mutation operations (inserts, updates, and deletes each add one to the dirt count) andmin_dirt_factoris the minimum dirt ratio (0.0 to 1.0). Both must be exceeded for compaction to trigger. -
NoAutoCompactDisable auto-compaction. Compaction can still be triggered manually with
compact.
Database configuration passed to open.
path is the directory where store files are kept (created if needed).
key_codec and value_codec control how keys and values are serialized
to bytes. Both must satisfy decode(encode(v)) == Ok(v) and produce the
same output for the same input.
key_compare is a total order over keys: deterministic, antisymmetric,
and transitive. It must agree with key_codec, so keys that compare
as Eq encode to identical bytes.
auto_compact toggles automatic compaction after writes. auto_file_sync
controls whether writes are automatically fsynced.
call_timeout is the number of milliseconds to wait for actor responses;
5000 is a reasonable starting point.
pub type Config(k, v) {
Config(
path: String,
key_codec: codec.Codec(k),
value_codec: codec.Codec(v),
key_compare: fn(k, k) -> order.Order,
auto_compact: AutoCompact,
auto_file_sync: FileSync,
call_timeout: Int,
)
}
Constructors
-
Config( path: String, key_codec: codec.Codec(k), value_codec: codec.Codec(v), key_compare: fn(k, k) -> order.Order, auto_compact: AutoCompact, auto_file_sync: FileSync, call_timeout: Int, )
An open database handle. Parameterized by key type k and value type v.
pub opaque type Db(k, v)
Controls whether writes are automatically fsynced to disk.
pub type FileSync {
AutoSync
ManualSync
}
Constructors
-
AutoSyncAutomatically fsync after every write for maximum durability.
-
ManualSyncDo not fsync automatically. Use
file_syncto flush manually.
A handle to a named keyspace. Obtained via trove.keyspace(...), which
registers the keyspace’s codecs and comparator with the database so later
operations (put_in, get_in, compaction) can operate on it.
pub opaque type Keyspace(k, v)
Errors that can occur when opening a database.
pub type OpenError {
DirectoryError(detail: String)
StoreError(detail: String)
LockError(detail: String)
ActorStartError
}
Constructors
-
DirectoryError(detail: String)The database directory could not be created or accessed.
-
StoreError(detail: String)The store file could not be opened or its header could not be recovered.
-
LockError(detail: String)The database path is already open by another actor on this node.
-
ActorStartErrorThe OTP actor failed to start.
A point-in-time snapshot handle for consistent reads.
pub type Snapshot(k, v) =
@internal Snapshot(k, v)
The result of a transaction callback. Return Commit to apply the
transaction’s writes, or Cancel to discard them.
pub type TransactionResult(k, v, a) {
Commit(tx: Tx(k, v), result: a)
Cancel(result: a)
}
Constructors
-
Commit(tx: Tx(k, v), result: a)Apply the transaction’s writes and return the result value.
-
Cancel(result: a)Discard the transaction’s writes and return the result value.
Values
pub fn close(db: Db(k, v)) -> Nil
Close the database and release the file handle and path lock. Does not
fsync. If using ManualSync, call file_sync before closing to ensure
durability. The Db handle must not be used after calling this.
Panics if the store file handle cannot be closed.
trove.close(db)
pub fn compact(
db: Db(k, v),
timeout timeout: Int,
) -> Result(Nil, String)
Trigger a manual compaction. Rebuilds the store file keeping only live
entries, resetting the dirt factor to zero. Returns Ok(Nil) on success
or Error(reason) if compaction failed. On failure the database remains
functional with the original store file.
The timeout is separate from call_timeout because compaction can take
much longer than normal operations.
let assert Ok(Nil) = trove.compact(db, timeout: 60_000)
pub fn delete(db: Db(k, v), key key: k) -> Nil
Remove a key. No error if the key does not exist.
Panics on store I/O errors (e.g. disk full, file corruption).
trove.delete(db, key: "hello")
pub fn delete_in(
db db: Db(k, v),
keyspace keyspace: Keyspace(k, v),
key key: k,
) -> Nil
Remove a key from a named keyspace. No error if the key does not exist.
Panics on store I/O errors.
trove.delete_in(db, keyspace: users, key: "alice")
pub fn delete_multi(db: Db(k, v), keys keys: List(k)) -> Nil
Atomically delete multiple keys.
Panics on store I/O errors (e.g. disk full, file corruption).
trove.delete_multi(db, keys: ["a", "b"])
pub fn delete_multi_in(
db db: Db(k, v),
keyspace keyspace: Keyspace(k, v),
keys keys: List(k),
) -> Nil
Atomically delete multiple keys from a named keyspace.
Panics on store I/O errors.
trove.delete_multi_in(db, keyspace: users, keys: ["alice", "bob"])
pub fn dirt_factor(db: Db(k, v)) -> Float
Returns the current dirt factor: a float between 0.0 and 1.0 that
approximates how much of the store file is occupied by superseded data.
Overwrites and deletes increment the dirt counter because they write new
nodes that make old ones unreachable. New inserts do not increment dirt
since they don’t supersede existing data. The formula is
dirt / (1 + size + dirt); the +1 ensures the result is always
well-defined, even for an empty tree. The value approaches but never
reaches 1.0. Higher values mean more wasted space that compaction would
reclaim.
let df = trove.dirt_factor(db)
pub fn file_sync(db: Db(k, v)) -> Nil
Force an fsync of the store file to disk. Useful when auto_file_sync
is set to ManualSync and you want to control when data is flushed.
Panics if the fsync system call fails.
let config = trove.Config(..config, auto_file_sync: trove.ManualSync)
let assert Ok(db) = trove.open(config)
trove.put(db, key: "hello", value: "world")
trove.file_sync(db)
pub fn get(db: Db(k, v), key key: k) -> Result(v, Nil)
Look up a key. Returns Ok(value) if found, Error(Nil) if the
key does not exist.
Panics on store I/O or decode errors (e.g. file corruption).
let assert Ok("world") = trove.get(db, key: "hello")
pub fn get_in(
db db: Db(k, v),
keyspace keyspace: Keyspace(k, v),
key key: k,
) -> Result(v, Nil)
Look up a key in a named keyspace. Returns Ok(value) if found,
Error(Nil) if the key does not exist.
Panics on store I/O or decode errors (e.g. file corruption, codec mismatch against on-disk bytes).
let assert Ok("admin") = trove.get_in(db, keyspace: users, key: "alice")
pub fn has_key(db: Db(k, v), key key: k) -> Bool
Check whether a key exists in the database.
Panics on store I/O or decode errors (e.g. file corruption).
let assert True = trove.has_key(db, key: "hello")
pub fn has_key_in(
db db: Db(k, v),
keyspace keyspace: Keyspace(k, v),
key key: k,
) -> Bool
Check whether a key exists in a named keyspace.
Panics on store I/O errors.
let assert True = trove.has_key_in(db, keyspace: users, key: "alice")
pub fn is_empty(db: Db(k, v)) -> Bool
Returns True if the database contains no entries.
let empty = trove.is_empty(db)
pub fn keyspace(
db db: Db(k, v),
name name: String,
key_codec key_codec: codec.Codec(k),
value_codec value_codec: codec.Codec(v),
key_compare key_compare: fn(k, k) -> order.Order,
) -> Keyspace(k, v)
Obtain a typed handle to a named keyspace. First use of a name registers it in this session; later uses update the codecs.
Panics if name collides with the reserved default-keyspace sentinel.
Codec trust model. Passing codecs that don’t match those previously
used for the same keyspace is undefined behavior: reads will likely
produce garbage values or panics. Keep the
(key_codec, value_codec, key_compare) tuple stable across opens for a
given keyspace name. Matches the trust model of Config.key_codec and
Config.value_codec.
let users =
trove.keyspace(
db,
name: "users",
key_codec: codec.string(),
value_codec: codec.string(),
key_compare: string.compare,
)
pub fn list_keyspaces(db db: Db(k, v)) -> List(String)
List the names of every keyspace currently registered on this database.
Returns names in sorted order. Includes every keyspace that has been
opened with trove.keyspace(...) in this session, plus every keyspace
that was persisted in the store file (even if not yet registered in this
session). Reading or writing a persisted-but-unregistered keyspace
without first calling trove.keyspace(...) panics.
let names = trove.list_keyspaces(db)
pub fn open(config: Config(k, v)) -> Result(Db(k, v), OpenError)
Open a database at the configured path. Creates the directory if it does not exist. If a store file already exists, recovers the tree from the latest valid header.
let config = trove.Config(
path: "./my_db",
key_codec: codec.string(),
value_codec: codec.string(),
key_compare: string.compare,
auto_compact: trove.AutoCompact(min_dirt: 1000, min_dirt_factor: 0.25),
auto_file_sync: trove.AutoSync,
call_timeout: 5000,
)
let assert Ok(db) = trove.open(config)
pub fn put(db: Db(k, v), key key: k, value value: v) -> Nil
Insert or update a key-value pair.
Panics on store I/O errors (e.g. disk full, file corruption).
trove.put(db, key: "hello", value: "world")
pub fn put_and_delete_multi(
db: Db(k, v),
puts puts: List(#(k, v)),
deletes deletes: List(k),
) -> Nil
Atomically insert and delete entries in a single operation. Puts are applied first, then deletes, all under a single header write.
Panics on store I/O errors (e.g. disk full, file corruption).
trove.put_and_delete_multi(
db,
puts: [#("new_key", "value")],
deletes: ["old_key"],
)
pub fn put_and_delete_multi_in(
db db: Db(k, v),
keyspace keyspace: Keyspace(k, v),
puts puts: List(#(k, v)),
deletes deletes: List(k),
) -> Nil
Atomically insert and delete entries in a named keyspace under a single header write. Puts are applied first, then deletes.
Panics on store I/O errors.
trove.put_and_delete_multi_in(
db,
keyspace: users,
puts: [#("bob", "admin")],
deletes: ["alice"],
)
pub fn put_in(
db db: Db(k, v),
keyspace keyspace: Keyspace(k, v),
key key: k,
value value: v,
) -> Nil
Insert or update a key-value pair in a named keyspace.
Panics on store I/O errors (e.g. disk full, file corruption).
trove.put_in(db, keyspace: users, key: "alice", value: "admin")
pub fn put_multi(
db: Db(k, v),
entries entries: List(#(k, v)),
) -> Nil
Atomically insert multiple key-value pairs. A single header write covers the entire batch.
Panics on store I/O errors (e.g. disk full, file corruption).
trove.put_multi(db, entries: [#("a", "1"), #("b", "2")])
pub fn put_multi_in(
db db: Db(k, v),
keyspace keyspace: Keyspace(k, v),
entries entries: List(#(k, v)),
) -> Nil
Atomically insert multiple key-value pairs into a named keyspace. A single header write covers the entire batch.
Panics on store I/O errors.
trove.put_multi_in(
db,
keyspace: users,
entries: [#("alice", "admin"), #("bob", "member")],
)
pub fn range(
db db: Db(k, v),
min min: option.Option(range.Bound(k)),
max max: option.Option(range.Bound(k)),
direction direction: range.Direction,
) -> List(#(k, v))
Iterate over entries in the database within optional key bounds.
Returns a List of key-value pairs.
For large result sets, use with_snapshot and snapshot_range instead
to stream entries lazily without loading them all at once.
Panics if the snapshot file handle cannot be opened, or on store read/decode errors during iteration.
Use range.Inclusive(key) or range.Exclusive(key) for bounds,
or option.None for unbounded. Use range.Forward or range.Reverse
for direction.
import gleam/option.{Some}
import trove/range
let results =
trove.range(
db,
min: Some(range.Inclusive("a")),
max: Some(range.Exclusive("z")),
direction: range.Forward,
)
pub fn range_in(
db db: Db(k, v),
keyspace keyspace: Keyspace(k, v),
min min: option.Option(range.Bound(k)),
max max: option.Option(range.Bound(k)),
direction direction: range.Direction,
) -> List(#(k, v))
Iterate over entries in a named keyspace within optional key bounds.
Returns a List of key-value pairs. For large result sets, use
with_snapshot and snapshot_range_in instead.
Panics if the keyspace has not been registered in this session via
trove.keyspace(...), if the snapshot file handle cannot be opened,
or on store read or decode errors during iteration.
let results = trove.range_in(
db,
keyspace: users,
min: Some(range.Inclusive("a")),
max: Some(range.Exclusive("z")),
direction: range.Forward,
)
pub fn set_auto_compact(
db: Db(k, v),
setting setting: AutoCompact,
) -> Nil
Change the auto-compaction setting at runtime.
trove.set_auto_compact(db, trove.AutoCompact(min_dirt: 1000, min_dirt_factor: 0.25))
pub fn size(db: Db(k, v)) -> Int
Returns the number of live entries in the database.
let count = trove.size(db)
pub fn size_in(
db db: Db(k, v),
keyspace keyspace: Keyspace(k, v),
) -> Int
Returns the number of live entries in a named keyspace.
let n = trove.size_in(db, keyspace: users)
pub fn snapshot_get(
snapshot snapshot: Snapshot(k, v),
key key: k,
) -> Result(v, Nil)
Look up a key in a snapshot. Returns Error(Nil) if the key does
not exist.
Panics on store read or decode errors (e.g. file corruption).
trove.with_snapshot(db, fn(snap) {
let assert Ok(value) = trove.snapshot_get(snapshot: snap, key: "my_key")
value
})
pub fn snapshot_get_in(
snapshot snapshot: Snapshot(k, v),
keyspace keyspace: Keyspace(k, v),
key key: k,
) -> Result(v, Nil)
Look up a key in a named keyspace within a snapshot.
Panics if the keyspace was not registered before the snapshot was acquired, or on store read or decode errors.
trove.with_snapshot(db, fn(snap) {
trove.snapshot_get_in(snap, keyspace: users, key: "alice")
})
pub fn snapshot_range(
snapshot snapshot: Snapshot(k, v),
min min: option.Option(range.Bound(k)),
max max: option.Option(range.Bound(k)),
direction direction: range.Direction,
) -> yielder.Yielder(#(k, v))
Iterate over entries in a snapshot within optional key bounds.
Returns a lazy Yielder that streams entries from disk on demand,
reading only one leaf node at a time.
The yielder holds a reference to the snapshot’s file handle, so it
must be consumed before the snapshot is closed. For large ranges,
prefer this over range to avoid loading all entries into memory.
Panics on store read or decode errors during iteration (e.g. file corruption).
Use range.Inclusive(key) or range.Exclusive(key) for bounds,
or option.None for unbounded. Use range.Forward or range.Reverse
for direction.
import gleam/option.{None, Some}
import gleam/yielder
import trove/range
let entries = trove.with_snapshot(db, fn(snap) {
let y = trove.snapshot_range(
snapshot: snap,
min: Some(range.Inclusive("a")),
max: None,
direction: range.Forward,
)
yielder.to_list(y)
})
pub fn snapshot_range_in(
snapshot snapshot: Snapshot(k, v),
keyspace keyspace: Keyspace(k, v),
min min: option.Option(range.Bound(k)),
max max: option.Option(range.Bound(k)),
direction direction: range.Direction,
) -> yielder.Yielder(#(k, v))
Iterate over entries in a named keyspace within a snapshot. Returns a
lazy Yielder streaming entries from disk.
The yielder holds a reference to the snapshot’s file handle; consume it before the snapshot closes.
Panics if the keyspace was not registered before the snapshot was acquired, or on store read or decode errors during iteration.
trove.with_snapshot(db, fn(snap) {
trove.snapshot_range_in(
snap,
keyspace: users,
min: Some(range.Inclusive("a")),
max: None,
direction: range.Forward,
)
|> yielder.to_list
})
pub fn transaction(
db: Db(k, v),
timeout timeout: Int,
callback callback: fn(Tx(k, v)) -> TransactionResult(k, v, a),
) -> a
Run an atomic transaction. The callback receives a Tx handle and must
return Commit(tx:, result: value) to apply writes or
Cancel(result: value) to discard. The transaction holds exclusive
write access for its duration.
The timeout parameter (in milliseconds) controls how long the caller
waits for the transaction to complete, including queue wait time and
callback execution. Choose a value appropriate for your workload.
Queued operations or auto-compaction may delay the start, and a
long-running callback consumes the remaining budget.
Important: The callback runs inside the database actor. Do not call
any trove functions (such as get, put, compact, etc.) on the
same Db handle from within the callback; this will deadlock the actor
until the call timeout fires. Use the Tx handle (tx_get, tx_put,
tx_delete) for all reads and writes inside the transaction.
Panics if the Commit variant contains a stale or replaced Tx
handle (e.g. the original handle instead of the latest one returned by
tx_put/tx_delete).
Non-escaping: The Tx handle is only valid inside the callback.
Do not store it in a variable, send it to another process, or return it.
Using a Tx after the callback returns will panic or produce undefined
behavior.
Timeout semantics: If the timeout fires while the callback is still executing, the caller panics but the actor continues running the callback to completion. This means writes may be durably committed even though the caller observes a timeout failure. Choose a timeout that accommodates your expected callback duration and any queued operations ahead of it.
let result = trove.transaction(db, timeout: 5000, callback: fn(tx) {
let tx = trove.tx_put(tx, key: "key", value: "value")
trove.Commit(tx:, result: "done")
})
pub fn tx_delete(tx tx: Tx(k, v), key key: k) -> Tx(k, v)
Delete a key within a transaction. Returns the updated Tx.
Panics on store I/O errors (e.g. disk full, file corruption).
trove.transaction(db, timeout: 5000, callback: fn(tx) {
let tx = trove.tx_delete(tx, key: "old_key")
trove.Commit(tx:, result: Nil)
})
pub fn tx_delete_in(
tx tx: Tx(k_default, v_default),
keyspace keyspace: Keyspace(k, v),
key key: k,
) -> Tx(k_default, v_default)
Delete a key from a named keyspace within a transaction.
Panics if the keyspace has not been registered in this session via
trove.keyspace(...), or on store I/O errors.
pub fn tx_get(tx tx: Tx(k, v), key key: k) -> Result(v, Nil)
Read a key within a transaction. Sees writes made earlier in the same
transaction. Returns Error(Nil) if the key does not exist.
Panics on store I/O or decode errors (e.g. file corruption).
trove.transaction(db, timeout: 5000, callback: fn(tx) {
let assert Ok(current) = trove.tx_get(tx, key: "counter")
let tx = trove.tx_put(tx, key: "counter", value: current <> "!")
trove.Commit(tx:, result: Nil)
})
pub fn tx_get_in(
tx tx: Tx(k_default, v_default),
keyspace keyspace: Keyspace(k, v),
key key: k,
) -> Result(v, Nil)
Look up a key in a named keyspace within a transaction. Sees writes made earlier in the same transaction.
Panics if the keyspace has not been registered in this session via
trove.keyspace(...), or on store I/O or decode errors.
pub fn tx_has_key(tx tx: Tx(k, v), key key: k) -> Bool
Check whether a key exists within a transaction. Sees writes made earlier in the same transaction.
Panics on store I/O or decode errors (e.g. file corruption).
trove.transaction(db, timeout: 5000, callback: fn(tx) {
let exists = trove.tx_has_key(tx, key: "counter")
trove.Commit(tx:, result: exists)
})
pub fn tx_has_key_in(
tx tx: Tx(k_default, v_default),
keyspace keyspace: Keyspace(k, v),
key key: k,
) -> Bool
Check whether a key exists in a named keyspace within a transaction.
Panics if the keyspace has not been registered in this session via
trove.keyspace(...), or on store I/O or decode errors.
pub fn tx_put(
tx tx: Tx(k, v),
key key: k,
value value: v,
) -> Tx(k, v)
Write a key-value pair within a transaction. Returns the updated Tx.
Panics on store I/O errors (e.g. disk full, file corruption).
trove.transaction(db, timeout: 5000, callback: fn(tx) {
let tx = trove.tx_put(tx, key: "greeting", value: "hello")
trove.Commit(tx:, result: Nil)
})
pub fn tx_put_in(
tx tx: Tx(k_default, v_default),
keyspace keyspace: Keyspace(k, v),
key key: k,
value value: v,
) -> Tx(k_default, v_default)
Insert or update a key-value pair in a named keyspace within a
transaction. Returns the updated Tx.
Panics if the keyspace has not been registered in this session via
trove.keyspace(...), or on store I/O errors.
pub fn with_snapshot(
db: Db(k, v),
callback callback: fn(Snapshot(k, v)) -> a,
) -> a
Run a callback with a point-in-time snapshot. The snapshot sees the state of the database at the moment it was acquired; subsequent writes are invisible to it.
Non-escaping: The Snapshot handle is only valid inside the callback.
Do not store it in a variable, send it to another process, or return it.
Using a Snapshot after the callback returns will panic or produce
undefined behavior because the underlying file handle is closed on exit.
Panics if the snapshot file handle cannot be opened.
let result = trove.with_snapshot(db, fn(snap) {
trove.snapshot_get(snapshot: snap, key: "my_key")
})
// result: Result(String, Nil)