File-based storage backend for raw time series segments and tier chunks.
Replaces SQLite shard tables with immutable .seg files for raw data
and append-only chunks.dat files with ETS indexes for tier data.
File Layout
shard_N/
raw/
1706000000.seg # immutable segment file per window
current.wal # rewritten on each flush with latest in-progress data
tier_hourly/
chunks.dat # append-only chunk data (same entry format as WAL)
index.ets # persisted ETS table for fast lookups
tier_daily/
chunks.dat
index.ets.seg File Format
┌─────────────────────────────────────┐
│ "TS" (2) | version (1) | count (4) │ 15 bytes header
│ index_offset (8) │
├─────────────────────────────────────┤
│ segment blobs (concatenated) │ variable
├─────────────────────────────────────┤
│ index entries (40 bytes each) │ count * 40
│ series_id:i64, start:i64, end:i64 │
│ point_count:u32, offset:u64, len:u32│
└─────────────────────────────────────┘WAL / chunks.dat Entry Format
series_id:i64 | start:i64 | end:i64 | count:u32 | data_len:u32 | data:bytesSummary
Functions
Delete all tier ETS tables (for cleanup on shutdown).
Delete the watermark ETS table (for cleanup on shutdown).
Compact a tier's chunks.dat by rewriting only live entries.
Delete raw .seg files and WAL entries before cutoff.
Delete tier chunks where chunk_end < cutoff. Removes from ETS index (dead space in chunks.dat).
Get distinct series_ids across raw .seg files and WAL.
Initialize shard storage directory.
Initialize tier storage for a specific tier.
Initialize watermark storage for this shard.
Merge two raw cache/WAL entries that share the same {series_id, start_time} key.
Persist all tier ETS indexes to disk.
Persist all watermarks to binary file.
Read ALL entries from all .seg files and WAL. Used for startup cache load.
Read raw segments for ALL series within a time range (for rollup).
Read the latest raw segment for a series.
Read raw segments for a specific series within a time range.
Read a single tier chunk by exact key (for rollup read-modify-write merge).
Read tier chunks for ALL series within a time range (for tier-to-tier rollup).
Read the latest tier chunk for a series.
Read tier chunks for a specific series within a time range.
Read a watermark value. Lock-free (ETS lookup).
Seal a completed time window: move matching WAL entries into an immutable .seg file.
Get aggregate stats for raw segments.
Calculate dead bytes in a tier's chunks.dat file.
Get distinct series_ids from a tier.
Get aggregate stats for a tier: {chunks, buckets, compressed_bytes}.
Write a batch of tier chunks. Appends blobs to chunks.dat and updates ETS index.
Write/update in-progress segments to the WAL.
Write a watermark value. Updates ETS and persists to file.
Functions
Delete all tier ETS tables (for cleanup on shutdown).
Delete the watermark ETS table (for cleanup on shutdown).
Compact a tier's chunks.dat by rewriting only live entries.
Reads live entries from ETS index, writes a new compact file, atomically renames it over the old file, then updates ETS offsets.
Options
:threshold- minimum dead_bytes/total_bytes ratio to trigger (default: 0.3)
Returns {:ok, reclaimed_bytes} or :noop if below threshold or empty.
Delete raw .seg files and WAL entries before cutoff.
Delete tier chunks where chunk_end < cutoff. Removes from ETS index (dead space in chunks.dat).
Get distinct series_ids across raw .seg files and WAL.
Initialize shard storage directory.
Initialize tier storage for a specific tier.
Creates the tier directory, loads or rebuilds the ETS index from chunks.dat, and returns the updated store.
Initialize watermark storage for this shard.
Creates an ETS table for lock-free reads and loads persisted values
from watermarks.bin. Missing tiers default to 0.
Merge two raw cache/WAL entries that share the same {series_id, start_time} key.
Returns a single {series_id, start_time, end_time, count, blob} tuple with
merged points, reusing the same logic as WAL/segment result merging.
Persist all tier ETS indexes to disk.
Persist all watermarks to binary file.
Read ALL entries from all .seg files and WAL. Used for startup cache load.
Read raw segments for ALL series within a time range (for rollup).
Read the latest raw segment for a series.
Read raw segments for a specific series within a time range.
Read a single tier chunk by exact key (for rollup read-modify-write merge).
Returns the blob binary, or nil if not found.
Read tier chunks for ALL series within a time range (for tier-to-tier rollup).
Returns {:ok, [[series_id, data], ...]} sorted by (series_id, chunk_start).
Read the latest tier chunk for a series.
Returns {:ok, [[data]]} or {:ok, []}.
Read tier chunks for a specific series within a time range.
Returns {:ok, [[data], ...]} sorted by chunk_start.
Read a watermark value. Lock-free (ETS lookup).
Seal a completed time window: move matching WAL entries into an immutable .seg file.
Get aggregate stats for raw segments.
Calculate dead bytes in a tier's chunks.dat file.
Dead bytes are bytes occupied by overwritten entries (from read-modify-write during rollup) that are no longer referenced by the ETS index.
Returns {dead_bytes, total_file_bytes}.
Get distinct series_ids from a tier.
Get aggregate stats for a tier: {chunks, buckets, compressed_bytes}.
Write a batch of tier chunks. Appends blobs to chunks.dat and updates ETS index.
entries = [{series_id, chunk_start, chunk_end, bucket_count, blob}, ...]
Write/update in-progress segments to the WAL.
Merges with existing WAL entries (same series_id+start_time → latest wins). Written atomically via tmp file + rename.
Write a watermark value. Updates ETS and persists to file.