Getting Started

View Source

The Erlang Rocksdb binding provides access to most of the key/value API of the rocksdb library.

Installation

Download the sources from our Github repository

To build the application simply run 'rebar3 compile'.

Note: cmake>=3.12 is required to install erlang-rocksdb.

To run tests run 'rebar3 eunit'. To generate doc, run 'rebar3 ex_doc'.

Or add it to your rebar config

{deps, [
    ....
    {rocksdb, "3.1.2"}
]}.

Or your mix config file:

{:rocksdb, "~> 3.1"}

Basic operations

Open a database


Path = "/tmp/erocksdb.fold.test",
Options = [{create_if_missing, true}],
{ok, DB} = rocksdb:open(Path, Options).

This gives you a nif resource, the database handle.

Keep the handle reachable for as long as you need the database: hold it in a live process, in ETS or in persistent_term. Once nothing references it any more, it is garbage collected, which closes the database and everything tied to it (iterators, column families). See OTP Integration for where to keep it in an application.

Closing a database

When you are done with a database run the following function

ok = rocksdb:close(DB).

The database is also closed for you once nothing holds the handle any more.

Read and Writes

You modify and query the database with put, get and delete. Each of them takes read or write options as its last argument, so pass [] when you have none. This writes a key, reads it back, then deletes it:

ok = rocksdb:put(Db, <<"my key">>, <<"my value">>, []),
case rocksdb:get(Db, <<"my key">>, []) of
  {ok, Value} -> io:format("retrieved value ~p~n", [Value]);
  not_found -> io:format("value not found~n", []);
  Error -> io:format("operational problem encountered: ~p~n", [Error])
end,
ok = rocksdb:delete(Db, <<"my key">>, []).

Batch Reads with multi_get

When you need to retrieve multiple keys at once, multi_get/3 is more efficient than calling get/3 multiple times. It retrieves all keys in a single operation:

%% Insert some data
ok = rocksdb:put(Db, <<"key1">>, <<"value1">>, []),
ok = rocksdb:put(Db, <<"key2">>, <<"value2">>, []),
ok = rocksdb:put(Db, <<"key3">>, <<"value3">>, []),

%% Retrieve multiple keys at once
Keys = [<<"key1">>, <<"key2">>, <<"key3">>, <<"missing">>],
Results = rocksdb:multi_get(Db, Keys, []),
%% Results = [{ok, <<"value1">>}, {ok, <<"value2">>}, {ok, <<"value3">>}, not_found]

The results are returned in the same order as the input keys. Each result is either:

  • {ok, Value} - the key was found
  • not_found - the key does not exist
  • {error, Reason} - an error occurred

For column families, use multi_get/4:

Results = rocksdb:multi_get(Db, ColumnFamily, Keys, []).

You can also use snapshots with multi_get to get a consistent view:

{ok, Snapshot} = rocksdb:snapshot(Db),
Results = rocksdb:multi_get(Db, Keys, [{snapshot, Snapshot}]),
rocksdb:release_snapshot(Snapshot).

Atomic Updates

If your process dies between two writes, you can be left with the same value stored under several keys. Collect the writes in a batch and apply them in one atomic operation instead:

{ok, Batch} = rocksdb:batch(),
ok = rocksdb:batch_put(Batch, <<"key1">>, <<"value1">>),
ok = rocksdb:batch_delete(Batch, <<"keytodelete">>),
ok = rocksdb:write_batch(Db, Batch, []),
ok = rocksdb:release_batch(Batch).

See the Batch API guide for savepoints, rollbacks and the other batch operations.

Synchronous writes

Using the setting {sync, true} in any write operations will makes sure to store synchronously on the filesystem your data.

Iterate your data

You can start an iterator using the function rocksdb:iterator/2 :

ItrOptions = [],
{ok, Itr} = rocksdb:iterator(DB, ItrOptions)

You close the iterator using the function rocksdb:iterator_close/1:

rocksdb:iterator_close(Itr)

You move the iterator with rocksdb:iterator_move/2:

  • move to the next key: rocksdb:iterator_move(Itr, next)
  • move to the previous key: rocksdb:iterator_move(Itr, prev)
  • start at the first key: rocksdb:iterator_move(Itr, first)
  • start at the last key: rocksdb:iterator_move(Itr, last)
  • move to a key: rocksdb:iterator_move(Itr, Key) or rocksdb:iterator_move(Itr,{seek, Key})
  • move to the previous key of: rocksdb:iterator_move(Itr,{seek_for_prev, Key})

An iterator doesn't support parallel operations, so make sure to use it accordingly.

You can optimize prefix seeks by putting your database in "prefix seek" mode with the prefix_extractor option on the database or the column family. See Prefix Seek for more information.

Wide-Column Entities

Starting in version 2.0.0, Erlang RocksDB supports RocksDB's Wide-Column Entity API. Entities allow storing structured data with multiple named columns per key, providing a more flexible schema than simple key-value pairs.

%% Store an entity with multiple columns
Key = <<"user:123">>,
Columns = [
    {<<"name">>, <<"Alice">>},
    {<<"email">>, <<"alice@example.com">>},
    {<<"age">>, <<"30">>}
],
ok = rocksdb:put_entity(Db, Key, Columns, []),

%% Retrieve entity columns as a proplist
{ok, Result} = rocksdb:get_entity(Db, Key, []),
Name = proplists:get_value(<<"name">>, Result),  %% <<"Alice">>

%% Delete entity (same as regular delete)
ok = rocksdb:delete_entity(Db, Key, []).

When iterating, you can access columns for any entry:

{ok, Itr} = rocksdb:iterator(Db, []),
{ok, Key, _Value} = rocksdb:iterator_move(Itr, first),
{ok, Columns} = rocksdb:iterator_columns(Itr),
%% For regular key-values, returns [{<<>>, Value}] (single default column)
%% For entities, returns all stored columns
rocksdb:iterator_close(Itr).

See Wide-Column Entities for more details.

Snapshots

Snapshots give you a consistent read-only view over the entire state of the key-value store. Set {snapshot, Snapshot} in the read options to make a read operate on that version of the database state.

{ok, Snapshot} = rocksdb:snapshot(Db),
ReadOptions = [{snapshot, Snapshot}],
{ok, Itr} = rocksdb:iterator(Db, ReadOptions),
... read using iter to view the state when the snapshot was created ...
rocksdb:iterator_close(Itr),
rocksdb:release_snapshot(Snapshot)

Note that when a snapshot is no longer needed, it should be released using the rocksdb:release_snapshot/1 function. This allows the implementation to get rid of state that was being maintained just to support reading as of that snapshot.

Column Families

Column Families provide a way to logically partition the database. Users can provide atomic writes of multiple keys across multiple column families and read a consistent view from them.

Backup and Checkpoint

Backup allows users to create periodic incremental backups in a remote file system (think about HDFS or S3) and recover from any of them.

Checkpoints provides the ability to take a snapshot of a running RocksDB database in a separate directory. Files are hardlinked, rather than copied, if possible, so it is a relatively lightweight operation.

Merge Operator

Starting in version 0.21.0, Erlang Rocksdb supports a Merge Operator for Erlang data types. A Bitset Merge Operator and a Counter Merge Operator are also available to change a bit at a position in a string or maintain a simple counter.

Transactions

Erlang RocksDB has preliminary support of multi-operation transactions. See Transactions

SST File Operations

You can work directly with SST (Sorted String Table) files, the storage format RocksDB uses, for bulk data loading, offline inspection and data migration.

Creating SST Files

Use sst_file_writer to create SST files outside the database:

%% Create an SST file
{ok, Writer} = rocksdb:sst_file_writer_open([], "/tmp/data.sst"),

%% Add key-value pairs (MUST be in sorted order)
ok = rocksdb:sst_file_writer_put(Writer, <<"key1">>, <<"value1">>),
ok = rocksdb:sst_file_writer_put(Writer, <<"key2">>, <<"value2">>),
ok = rocksdb:sst_file_writer_put(Writer, <<"key3">>, <<"value3">>),

%% Finish and get file info
{ok, FileInfo} = rocksdb:sst_file_writer_finish(Writer, with_file_info),
ok = rocksdb:release_sst_file_writer(Writer),

%% FileInfo contains: file_path, smallest_key, largest_key,
%% file_size, num_entries, sequence_number

Ingesting SST Files

Load SST files directly into a database:

{ok, Db} = rocksdb:open("/tmp/mydb", [{create_if_missing, true}]),

%% Ingest SST files into the database
ok = rocksdb:ingest_external_file(Db, ["/tmp/data.sst"], []),

%% Data is now queryable
{ok, <<"value1">>} = rocksdb:get(Db, <<"key1">>, []),

%% Ingest into a specific column family
ok = rocksdb:ingest_external_file(Db, ColumnFamily, ["/tmp/cf_data.sst"], []),

rocksdb:close(Db).

Reading SST Files

Inspect SST files without loading them into a database:

%% Open an SST file for reading
{ok, Reader} = rocksdb:sst_file_reader_open([], "/tmp/data.sst"),

%% Get table properties (metadata)
{ok, Props} = rocksdb:sst_file_reader_get_table_properties(Reader),
io:format("Entries: ~p, Size: ~p bytes~n",
    [maps:get(num_entries, Props), maps:get(data_size, Props)]),

%% Iterate through the file
{ok, Itr} = rocksdb:sst_file_reader_iterator(Reader, []),
{ok, Key1, Value1} = rocksdb:sst_file_reader_iterator_move(Itr, first),
{ok, Key2, Value2} = rocksdb:sst_file_reader_iterator_move(Itr, next),

%% Seek to a specific key
{ok, Key, Value} = rocksdb:sst_file_reader_iterator_move(Itr, {seek, <<"key2">>}),

%% Verify file integrity
ok = rocksdb:sst_file_reader_verify_checksum(Reader),

%% Cleanup
ok = rocksdb:sst_file_reader_iterator_close(Itr),
ok = rocksdb:release_sst_file_reader(Reader).

See SST Files Guide for more details including options, best practices, and advanced use cases.