Optional TTL cache for inference results.

Off by default, and deliberately so. Caching a prediction is only correct when the same input must produce the same output — which is false the moment a model reads a clock, a random seed, or mutable feature state. Silently caching by default would turn that into a subtle correctness bug, so MLServe.predict/3 caches only when you ask:

MLServe.predict(:fraud_detection, features, cache: true)

or per model:

config :ml_serve,
  models: [fraud_detection: [cache: [enabled: true, ttl: :timer.minutes(1)]]]

Design

A single :public ETS table with write_concurrency: true. Readers and writers work directly in the calling process — the GenServer exists only to own the table and run the sweeper, and is never in the request path.

Expiry is both lazy (checked on read) and swept (a periodic :ets.select_delete). Lazy alone leaks memory for keys never read again; sweeping alone lets a reader see a stale entry between sweeps.

Keys

The default key is :erlang.term_to_binary(input, [:deterministic]) hashed with SHA-256. The :deterministic flag matters: without it, large maps serialise in internal-hash order and two equal inputs can produce different binaries, silently halving the hit rate.

For large tensor inputs, hashing the whole term is more expensive than the inference you are trying to skip. Pass a cheap key instead:

MLServe.predict(:fraud, features, cache: true, cache_key: features.account_id)

What is not cached

Errors are never cached — a transient backend failure must not be pinned for the TTL.

Eviction

TTL plus a :max_size bound, enforced at sweep time. There is no LRU: tracking recency needs a write on every read, which would put a serialised write on the hot path to save memory that a TTL already bounds. If you need LRU, use Cachex and pass results through :postprocess.