erllama_cache (erllama v0.11.0)
View SourcePublic API of the KV cache: tier setup, inspection, eviction and counters.
The RAM tier is always on. Add a disk or ram_file tier with
add_tier/1 (or the tiers application environment key) and point a
model at it with tier + tier_srv in its load config:
ok = erllama_cache:add_tier(#{name => kv_disk, backend => disk,
root => "/var/lib/erllama/kv"}),
{ok, M} = erllama:load_model(#{model_path => Path,
tier => disk, tier_srv => kv_disk}).Tier servers, the meta server, the writer pool and the policy module
are internal; the runtime plumbing lives in erllama_cache_meta_srv
and erllama_cache_writer.
Summary
Functions
Start a supervised disk or ram_file tier server registered as
name and rooted at root (created if missing). Models reference it
with tier_srv => name and tier => backend.
Evict oldest available rows until at least TargetBytes have been
freed. Returns {evicted, NumRows, BytesFreed}.
Like evict_bytes/1, but only considers rows whose tier is in
Tiers. Pass all to match every tier, or a subset of
[ram, ram_file, disk]. The system-pressure scheduler uses this to
evict only RAM-resident slabs while leaving the disk tier alone.
Synchronous full eviction pass. Walks the LRU and drops every available row with refcount=0. Returns the number evicted.
Snapshot of operational counters as a map of slot name to non-negative integer. Suitable for forwarding to a metrics exporter (Prometheus, statsd, OpenTelemetry).
Snapshot of the cache: the tiers, the number of cached rows and the bytes held per tier backend.
Every tier server, the built-in RAM tier included.
Find the longest cached rendered-byte prefix of PromptBytes for
the given key namespace (#{fingerprint, quant_type, ctx_params_hash}). Content-addressed (ds4-style): returns
{ok, MatchBytes, Row} where MatchBytes is the byte length of the
longest stored prompt prefix, or miss. Operator-friendly entry
point for stateless callers (HTTP front-end, agent loops) that
resend the full conversation each turn.
Stop a tier started with add_tier/1. Rows saved in it become unavailable.
Reset all counters to 0. Mostly for tests; production callers should treat counters as monotonic-since-boot.
Types
-type cache_key() :: <<_:256>>.
-type save_reason() :: cold | continued | finish | evict | shutdown.
-type status() :: available | writing | evicting.
-type tier() :: ram | ram_file | disk.
-type tier_info() :: #{name := atom(), backend := tier(), root := file:name() | undefined, pid := pid()}.
One entry of list_tiers/0.
Spec for add_tier/1 and the tiers environment key.
Functions
Start a supervised disk or ram_file tier server registered as
name and rooted at root (created if missing). Models reference it
with tier_srv => name and tier => backend.
-spec evict_bytes(non_neg_integer()) -> {evicted, non_neg_integer(), non_neg_integer()}.
Evict oldest available rows until at least TargetBytes have been
freed. Returns {evicted, NumRows, BytesFreed}.
-spec evict_bytes(non_neg_integer(), all | [tier()]) -> {evicted, non_neg_integer(), non_neg_integer()}.
Like evict_bytes/1, but only considers rows whose tier is in
Tiers. Pass all to match every tier, or a subset of
[ram, ram_file, disk]. The system-pressure scheduler uses this to
evict only RAM-resident slabs while leaving the disk tier alone.
-spec gc() -> {evicted, non_neg_integer()}.
Synchronous full eviction pass. Walks the LRU and drops every available row with refcount=0. Returns the number evicted.
-spec get_counters() -> #{atom() => non_neg_integer()}.
Snapshot of operational counters as a map of slot name to non-negative integer. Suitable for forwarding to a metrics exporter (Prometheus, statsd, OpenTelemetry).
-spec info() -> #{tiers := [tier_info()], entries := non_neg_integer(), bytes := #{ram := non_neg_integer(), ram_file := non_neg_integer(), disk := non_neg_integer()}}.
Snapshot of the cache: the tiers, the number of cached rows and the bytes held per tier backend.
-spec list_tiers() -> [tier_info()].
Every tier server, the built-in RAM tier included.
-spec lookup_longest_text_prefix(map(), binary()) -> {ok, non_neg_integer(), tuple()} | miss.
Find the longest cached rendered-byte prefix of PromptBytes for
the given key namespace (#{fingerprint, quant_type, ctx_params_hash}). Content-addressed (ds4-style): returns
{ok, MatchBytes, Row} where MatchBytes is the byte length of the
longest stored prompt prefix, or miss. Operator-friendly entry
point for stateless callers (HTTP front-end, agent loops) that
resend the full conversation each turn.
-spec remove_tier(atom()) -> ok | {error, not_found}.
Stop a tier started with add_tier/1. Rows saved in it become unavailable.
-spec reset_counters() -> ok.
Reset all counters to 0. Mostly for tests; production callers should treat counters as monotonic-since-boot.