Changelog

View Source

All notable changes to erllama are documented here. The format follows Keep a Changelog and this project adheres to Semantic Versioning.

[Unreleased]

[0.10.1] - 2026-08-23

Fixed

  • chat/3 and chat_apply/3 hand tools to llama.cpp in the OpenAI shape ({"type": "function", "function": {...}}); the flat chat_tool() maps were rejected with chat_parse_failed.
  • erllama_chat_SUITE starts the application with its dependencies when run on its own.

Added

  • Load option chat_template: Jinja source replacing the template stored in the GGUF for chat/3 / chat_apply/3.
  • examples/agent_loop: a tool-calling agent loop on chat/3.

[0.10.0] - 2026-08-23

Changed

  • Vendored llama.cpp bumped from b10068 to b10593 (vendor/ is now kept whole: upstream builds it as CMake targets). model_opts gains load_mode (upstream's llama_load_mode); use_mmap / use_mlock are mapped onto it.

Removed (BREAKING)

  • unload_model/1 (use unload/1), models/0 (use list_models/0), list_cached_prefixes/2 (renamed cached_prefix_len/2).
  • infer/4: use stream/3 (text or tokens; the receiving process is the to option, default the caller). continue/3 takes to instead of caller_pid; a missing session_id is {error, {missing_option, session_id}}.
  • Stream messages {erllama_token, Ref, _}, {erllama_token_id, Ref, _}, {erllama_thinking_end, Ref, _}, {erllama_done, Ref, _}, {erllama_error, Ref, _}: every event is now {erllama, Ref, Event} with Event :: {token, Bin} | {token_id, Id} | {thinking, Bin} | {thinking_end, Sig} | {done, Stats} | {error, Reason} (erllama:stream_event()).
  • apply_chat_template/2 renamed render_chat_template/2.
  • chat_apply/2 is chat_apply/3 (model, messages, opts) and returns {ok, #{prompt, params}}; messages and tools are Erlang maps, JSON encoding happens at the NIF boundary.
  • verify/4 returns {ok, #{accepted, next}}.
  • set_observer/1 and clear_observer/0: use a middleware (erllama_middleware, guides/middleware.md).
  • Application environment: chat_params_cache_size renamed chat_cache_size; quota_mb dropped from the tiers entries.

Changed (BREAKING)

  • Every per-model call returns {ok, Result} or {error, not_loaded} for an unknown or stopped model instead of exiting with noproc: model_info/1, status/1, phase/1, pending_len/1, queue_depth/1, last_cache_hit/1, list_adapters/1 now wrap their result in {ok, _}; unload/1, evict/1, shutdown/1, end_session/2 return {error, not_loaded}.
  • load_model/1,2 validates the config (erllama_opts): backend defaults to erllama_model_llama; a missing model_path is {error, {missing_config, model_path}}, a missing file is {error, {invalid_config, model_path, Path}}, an unknown key is {error, {unknown_option, Key}}. model_id in the config map is honoured by load_model/1.
  • complete/3, prefill_only/3 and infer/4 validate their option maps: unknown keys are {error, {unknown_option, Key}}, wrong types {error, {invalid_option, Key, Value}}.
  • response_tokens defaults to 64 on every path (complete/3 used 4).
  • evict/1 and shutdown/1 honour the evict_save_timeout_ms application environment key (default 30 s); it was documented but unread.
  • list_adapters/1 entries use the key adapter (was handle).

Added

  • erllama:stream/3 and erllama:collect/2: streaming inference with a typed event envelope and a collector that folds the events into a stream_result().
  • erllama:chat/3: one chat turn (render, generate, parse) with Erlang-term messages and tools; returns the parsed assistant message with content, reasoning and tool calls.
  • erllama:embed/2 accepts text; erllama:embed_batch/2 embeds a list of inputs in one round-trip to the model process.
  • erllama:whereis/1 returns the model pid for monitoring.
  • Supervised cache tiers: erllama_cache:add_tier/1, remove_tier/1, list_tiers/0, info/0, and the tiers application environment key ([#{name, backend => disk | ram_file, root}]) started with the application. load_model checks that tier_srv is running and matches tier.
  • erllama_middleware: hackney-style middleware chain around every API call (global via the middleware environment key, or per call with the middleware option).
  • erllama:pressure/0, pressure_sources/0, requests/0, request_info/1.
  • Application environment keys fingerprint_mode (now the default for models that do not set it), writer_max_concurrent, chat_cache_size, thinking_signing_key, middleware and tiers are declared in the app file and documented; os_mon is a declared dependency (the system pressure source needs memsup).
  • erllama_scheduler:validate_config/1 checks that model_evictor names a loadable module exporting evict_one/0.
  • Documentation: public modules are erllama, erllama_cache, erllama_middleware, erllama_scheduler and the erllama_model_backend, erllama_model_evictor, erllama_pressure behaviours plus the erllama_model_stub test backend; every other module is hidden from hexdocs. Guides rewritten around the public API (the tool-calls guide now documents chat/3; the tool_call_markers option it described never existed). Public types are defined in erllama.
  • Tests: shared erllama_test_helpers; no catch Expr left, so the suite compiles on OTP 29 without nowarn_deprecated_catch.
  • erllama exports the types its specs use (token_id/0, cache_key/0, completion_result/0, stats/0, request_opts/0, load_config/0, error_reason/0, ...) and documents every error reason in error_reason/0.

[0.9.0] - 2026-08-23

Added

  • erllama_chat:set_observer/1' /clearobserver/0' hook. Lets a separate module (typically the server's metrics module) observe wall-time of every chat-NIF call (apply / parse) without the runtime taking a compile-time dependency on the metrics module. Registered via persistent_term; unset = no-op. Drives the new `erllama_chat_duration_seconds' Prometheus histograms on the server side. - erllama_app:start/2' enableserlang:system_flag(scheduler_wall_time, true)' at boot so callers (the server's metrics module among them) can read per-scheduler busy ratios for the autoparser dirty-pool monitoring. ### Changed - chat_apply/2' runs upstream'scommon_chat_templates_apply' once per request (prompt + parser). The per-tools params cache and the render-only NIF are gone; only the per-model templates ref is cached. - Vendored llama.cpp is b10068, upstream and unmodified (no local patches). scripts/vendor_llama.sh <tag> performs bumps and fails if any vendored file differs from the tarball. Unused vendor/ libraries are pruned and LLAMA_OPENSSL is off (no OpenSSL link). - Hex package manifest now includes c_src/erllama_chat_nif.{cpp,h} and c_src/erllama_resources.h (previous tarballs could not build). - Bump vendored llama.cpp from b9334 to b9585. No API-breaking changes on our touchpoints; brings `common/chatbug fixes (LFM2 reasoning, tool-parser unification). UPDATE_LLAMA.md refreshed to document thecommon/+vendor/sync the prior procedure left implicit. ### Removed (BREAKING) - Streaming API:{erllamatoken, , {toolcall_delta, }}' and erllama_tool_call_end' messages, plus the matching step result variants. The engine no longer classifies tool-call bytes mid-stream. Callers usingerllama:infer/4' directly buffer tokens and call erllama:chat_parse/3' at done for structured extraction. HTTP wire format on the server is unchanged. - Backendtool_call_end_is_eos/1' callback.

Changed

  • erllama:chat_apply/3' becomeschat_apply/2' (drops the ToolsHash' parameter).erllama_chat_cache' shrinks to caching only the heavy common_chat_templates_init' ref per model; each request invokescommon_chat_templates_apply' fresh because the synthesized parser is sensitive to tool_choice' andparallel_tool_calls' (now folded into the NIF Inputs map).

Added

  • Public chat_apply/2 + chat_parse/3 that delegate to erllama_chat via the model gen_statem so callers can build a chat_params_ref' and parse model output without touching the underlying NIF model resource. Backend gains an optionalget_model_ref/1' callback; the stub backend returns {error, chat_not_supported}'.chat_purge/1' drops cached entries on demand for a given model id.

Fixed

  • EOS-bounded tool-call flush: don't capture the EOS token's bytes before flushing the span. The previous in-span EogFlag path called req_tool_call_emit/3 unconditionally; for Granite / Phi-4 the EOS is a special token that detokenizes empty under special=false, so the buffer stayed clean in practice, but the path was fragile for any future model whose EOS detokenises to visible bytes. Now the EOS-end branch skips the emit and calls req_tool_call_end/2 directly, matching the byte-string-end-marker semantics.

Added

  • Complete the chat-autoparser NIF wiring. nif_chat_templates_apply now translates an Erlang inputs map (messages + tools JSON binaries, optional tool_choice) into common_chat_templates_inputs via common_chat_msgs_parse_oaicompat / common_chat_tools_parse_oaicompat, calls upstream, and returns the synthesized chat_params_ref plus the rendered prompt bytes. nif_chat_parse deserialises the per-template PEG arena from the cached params string and dispatches to common_chat_parse. The parsed common_chat_msg is marshalled to #{role, content, reasoning_content, tool_calls}; tool-call arguments come back as raw JSON binaries and are decoded to maps at the Erlang facade boundary (erllama_chat:parse (arity 3)). New erllama_chat_SUITE real-model CT (gated on LLAMA_TEST_MODEL) covers init / apply / parse round-trip and a partial-then-full streaming case.
  • NIF wrapper for llama.cpp's common_chat_* autoparser. Vendors common/ + vendor/nlohmann + vendor/cpp-httplib from the pinned llama.cpp tree, flips LLAMA_BUILD_COMMON=ON, and links llama-common into the NIF .so. Three new entry points (nif_chat_templates_init/2, nif_chat_templates_apply/2, nif_chat_parse/3) run on dirty CPU; two new resources (chat_templates_ref, chat_params_ref) wrap common_chat_templates_ptr and common_chat_params. New Erlang facade erllama_chat (raw NIF shim) and erllama_chat_cache (LRU cache keyed on {ModelIdBin, ToolsHash}, with purge/1 for model unload). The refactored erllama_resources.h exposes the existing C resource type pointers + erllama_model_t to C++ TUs behind an extern "C" guard. templates_init works end-to-end; templates_apply and parse ship as {error, not_implemented} placeholders (the Erlang-term marshalling lands in the follow-up). Dormant capability: the chat / messages / responses handlers do not consume this surface yet; Phase 3.C wires them up.
  • EOS-bounded tool-call end-marker capture. Models configured with tool_call_markers => #{start => Bytes, 'end' => <<"$eos">>} opt into the new path: when the scheduler is inside an open tool-call span and the model samples a token with EogFlag = 1, the accumulated tool_call_bytes buffer is flushed via the existing erllama_tool_call_end message before the request finishes. Previously the buffer was silently dropped in that branch (the only emission site was the byte-string end-marker match in erllama_model_llama:map_marker (arity 2)). The byte-string-end families (Mistral </s>, Qwen </tool_call>, DeepSeek <｜tool▁call▁end｜>, Llama 3.1 <|eom_id|>) are byte-exact unchanged; the new path is opt-in via the sentinel binary <<"$eos">> on the end' key. Backend behaviour gains one optionaltoolcall_end_is_eos/1callback (the scheduler defaults tofalsefor backends that have not been updated). Targets the IBM Granite-3.x and Microsoft Phi-4-mini families whose wire shape bounds the call at EOS rather than a byte-string end marker; the server-side migrations land in follow-up PRs. ### Changed - The native tool-call capture splits spans on a repeated start marker. Families like Mistral tekken delimit parallel calls by emitting the start marker again ([TOOL_CALLS]n1[ARGS]a1[TOOL_CALLS]n2[ARGS]a2</s>) with no per-call end between them; the previous single-span capture concatenated every call into oneerllama_tool_call_endmessage, which the server parsed as one garbled call.apply_step_results/2now finalises the current span before opening the next when a start marker fires while already in-span, so each call yields its own end message and reuses the upstream per-call accumulation. Behaviour-preserving for families with explicit per-call end markers (qwen-xml, qwen3-coder, dsml): a start never fires while in-span because the end fires first. The stub backend gains an optionaltool_call_script :: [start | body | end_tok]config so tests can drive exact decode-step sequences (the Mistral tekken[start, body, body, start, body, body, end_tok]shape and the qwen-style[start, body, end_tok, start, body, end_tok]regression guard run from the same harness). - The tool-call end marker now carries the source token's eog flag.erllama_model_llama:map_marker(arity 2) was discardingEogFlagon the end marker, so a model whose end marker IS the eos token (Mistral tekken uses</s>as both the per-span end marker and the assistant turn's eos) kept decoding past the close and spammed repeated tool calls under the greedy continuation. Backend step results emit{tool_call_end, Eog}instead of the bare atom; the scheduler setsReq.finishing = trueon eog so the turn ends cleanly. Backend type spec updated accordingly. -erllama_model_stubgains an opt-instep_delay_ms :: non_neg_integer()test knob thattimer:sleeps for the configured number of milliseconds at the top of everystep/2call. Lets server-side concurrency tests deterministically keep a holder request in-flight while other requests race for the queue (used by the e2e suite'schat_busy_returns_429to fix a pre-existing CI timing flake where the holder's stream could finish before the racing requests arrived). Default 0 is a true no-op for the cache / integration tests that already use the stub. Invalid configs coerce to 0. ### Fixed - Idle sticky-session seq pins are now reclaimed under seq-pool pressure, so the pool no longer permanently exhausts. A completed sticky turn keeps its sequence pinned for warm continuation, but those pins were never released (onlyend_session/2or model stop freed them) - so aftern_seq_maxdistinct sessions, every new session got{error, seq_capacity}(529) with retries never recovering. Admission now reclaims the least-recently-used **idle** pin (asession_seqentry whose seq has no in-flight request) when the pool is full, on both the fresh-admit (admit_normal/2) and queued- dispatch (dispatch_pending_admits/2) paths, while never reclaiming an in-flight session.seq_capacitynow only fires when every seq is genuinely active. A reclaimed session re-admits cold (or warm-restores from the tiered cache) on its next turn.model_info/1gainspinned_idle_seqs(reclaimable headroom;available_seqsstays ~0 since an admitted request immediately re-pins). - Native tool-call capture now keeps the body between the markers. The marker scanner (erllama_model_llama:map_marker(arity 2)) is stateless - it tags only the start/end marker tokens - so for a model whose tool-call body is ordinary tokens (e.g. Qwen3-Coder's<tool_call><function=NAME><parameter=P>v</parameter></function></tool_call>) the body was streamed as content and the capturedtool_call_bytesheld only the start marker, yielding an empty call (name "unknown",arguments {}). The decode loop now accumulates any token sampled while a tool-call span is open (active_sampler=tool_call_syntax/tool_call_payload) into the tool-call bytes instead of the content stream, so the full call reaches the parser. Models that emit their call without the marker tokens (e.g. Qwen2.5qwen-xmlJSON) are unaffected - they never open a span. ### Added - Scheduler can proactively unload an idle model under sustained memory pressure (opt-inscheduler.unload_models_under_pressure => true). After cache eviction runs, if it could not free the target and pressure is still high, the scheduler calls the configuredmodel_evictor(newerllama_model_evictorbehaviour,evict_one/0) to unload the least-recently-active idle model. Cache slabs are always freed first; at most one model is unloaded per tick; the callback's return is validated so a missing or misbehaving evictor degrades to no-op.status/0reportsmodels_unloaded_total/last_model_unloaded. The engine names the evictor module via config only - no compile-time dependency on the server app that implements it. ### Changed - Vendored llama.cpp bumped fromb9222tob9334(ggml 0.12.0 -> 0.13.0). No erllama source changes required; the C ABI surfaces the NIF wraps (erllama_safe.cpp: model load/free, context, sampler chain,llama_decode,llama_memory_seq,llamastate_seq) are unchanged. - Cache eviction is now frecency-scored, not pure LRU. Byte-targeted eviction (evictbytes, scheduler memory pressure) drops the lowest-scoring rows first, where the score is recency biased forward by the row's hit count with a 6 h half-life decay (erllama_cache_meta_srv:eviction_score/3). Pinned static-prefix rows and the currently-live session key (set_live_key/1, ds4protected_sha) sort in a higher rank so they survivegc/0and are evicted byevict_bytesonly as a last resort when nothing else frees enough. The old one-shotlast_used += hits1sinstall bias is removed (hits now feed the score directly, so they are no longer double-counted). ### Added - Pinned static-prefix (agent_prefix) checkpoints. When a caller supplies a verifiedprefix_checkpoint_len(the end-of-tools token offset), the cold prefill writes anagent_prefixKV checkpoint at exactly that boundary - independent of thecold_min/cold_maxband - and pins it so the hot, shared system+tools prefix is not evicted under churn. The pin is bounded to one row per namespace (erllama_cache_key:namespace/3), survives restart (recovered from the persisted save reason on the disk scan), and is re-applied on a warm resume that lands on the boundary. Newerllama_cache_meta_srv:pin_row/2,?POS_PINNEDrow field,?C_SAVES_AGENT_PREFIXcounter (saves_agent_prefix), and KVC save reason 6 (mirrors ds4AGENT_SYSTEM). Cold-prefill segments now carry a per-boundary save reason. ### Changed - KV cache is now keyed by the **rendered prompt bytes** (detokenize(tokens)), not the token-id list (ds4-style, content-addressed). The same logical prompt now hits across turns even when it retokenises (chat-template wrapping, tool rendering, generated ids vs re-tokenised assistant text), where the old token-keyed cache went cold every turn. The longest-prefix lookup (erllama_cache_meta_srv:lookup_longest_text_prefix/2) scans stored byte-prefix lengths and resumes from the longest match. The uncovered suffix reuses the caller's original tokens when the byte boundary lands on a token boundary (token-exact resume - so a re-sent prompt still reproduces its reply); only a mid-token boundary re-tokenises the byte remainder (checkpoint_tokens ++ tokenize(byte_suffix), identical byte stream, sound). Hits stay exact (SHA-256 over the bytes; no fuzzy match). The KVC file format version is bumped to **v2**; old v1 (token-keyed) files are rejected on the startup disk scan, so the cache refills under the new scheme.list_cached_prefixes/2` now reports the matched *byte length. ### Added - Per-context compiled-grammar cache. An identical GBNF (agentic clients like Claude Code resend the same tool grammar every turn) is now parsed once and cloned per request via llama_sampler_clone instead of re-parsed, which dominated infer admission for large tool grammars. The cache is a small byte-verified LRU on the context resource (a hash is only a pre-filter; identity is confirmed by length + memcmp), freed with the context. New erllama_nif:grammar_cache_stats (arity 1) (#{hits, misses}) exposes whether the cache is taking effect. ## [0.8.0] - 2026-05-23 Engine-robustness release covering the erllama_server hardening brief observed under real 30B/Metal load (cold-admit decode wedges, agentic tool-continue loops). ### Added - Bounded, interruptible, self-recovering decode. Every context installs a ggml abort callback. Each decode step arms a per-step wall-clock budget (context_opts.decode_budget_ms, default 30000, 0 disables); exceeding it aborts the decode and returns {error, decode_timeout} instead of blocking forever. erllama_nif:request_abort (arity 1) sets an atomic flag the callback honours without taking the context mutex, so a running decode can be interrupted from outside the (blocked) gen_statem and returns {error, decode_aborted}; cancel/1 fires it best-effort. On decode_timeout/decode_aborted the engine recovers in place: fails in-flight and queued callers, recreates the context via the new backend reset_context/1 (model stays loaded), resets seq and session state, and returns to idle. Recovery drops only the live in-context KV cells and sticky-session pins; the persistent tiered cache (RAM/disk rows) is untouched, so the next admission still warm-restores from cache where a saved row matches rather than starting fully cold. - on_full => block | error admission option on complete/3, prefill_only/3, infer/4 (default block). error fails fast with {error, seq_capacity} instead of queueing when no seq is free — pair with available_seqs / n_seq_max from model_info/1. - generated => [token_id()] in the erllama_done Stats map: the exact generated token ids in order, so a caller can build a byte-exact suffix for continue/3 without re-tokenising detokenised text. - expect_committed => [token_id()] option on continue/3: the caller's view of the session's committed tokens. When supplied it must equal the stored context exactly, otherwise continue/3 returns {error, {transcript_mismatch, #{stored_len, expected_len, diverge_at}}} without prefilling, leaving the seq pinned for a re-sync and retry. ### Changed - A binary grammar is now authoritative through tool-call syntax tokens. Previously, on a model with tool_call_markers, a tool_choice=required / response_format grammar was abandoned once a tool-call span opened (syntax tokens went through a grammar-less greedy sampler), letting output drift to free-form. The greedy-on-syntax swap is now disabled for any request carrying a grammar, so the constraint holds end to end on infer/4 and continue/3. - nif_decode_one now returns {error, {decode_failed, Rc}} instead of a bare {error, Rc}, matching nif_step. ## [0.7.0] - 2026-05-20 ### Added - erllama:reset_session/2 recovery primitive. Forcibly drops a sticky session's live KV cells and any in-flight #req{} on its seq, then returns the seq slot to the idle pool. Uses a 5 s gen_statem:call timeout so it stays reachable when the engine's infinity-timeout hot path is wedged. Returns {ok, recovered | not_found} | {error, timeout}. Streaming callers on the reset seq receive {erllama_error, Ref, engine_reset}. - n_seq_max and available_seqs keys in erllama:model_info/1. available_seqs is the live idle-list length; sticky-pinned seqs count as unavailable. Lets callers detect saturation up front instead of inferring it from sticky_busy errors. ### Changed - nif_step now returns {error, {decode_failed, Rc}} (was bare {error, decode_failed}). The Rc integer surfaces the llama_decode return code (1, -1, 2, ...) so operators can tell OOM from KV-cache corruption from sample rejection. The exception path ({error, exception}) is unchanged. ## [0.6.2] - 2026-05-19 ### Changed - Vendored llama.cpp bumped from b9119 to b9222. No erllama source changes required; the C ABI surfaces we depend on (model load/free, context, sampler chain, llama_n_batch, `llama_state_seq) are unchanged. ## [0.6.1] - 2026-05-18 ### Fixed - BEAM segfault inapplychat_template/2on rendered chat-template output above the initial 4 KiB render buffer. The vendoredllama_chat_apply_templatereturns the full formatted size as a positive value even when the caller's buffer was too small (strncpysilently truncates). The NIF retry path only fired on negative return, so the positive size was fed astext_lentollama_tokenizeand the subsequentstd::stringconstruction walked past the truncated buffer into unmapped pages. Retry now triggers onwritten > buf_size. Caps bumped to match the downstream's 64 MiB body limit:ERLLAMA_MAX_TOKEN_TEXT4 MiB → 64 MiB,ERLLAMA_MAX_TOKENS1 M → 16 M. The token output cap is enforced on the tokenize success path so byte-fallback tokenizers don't return over-cap lists. ## [0.6.0] - 2026-05-18 ### Added -erllama:continue/3for caller-asserted chat-template continuation on a sticky session. Skips the prompt prefix-equality check inresolve_sticky_continuationso chat templates whose rendered prefix shifts between turns can still reuse the live KV cells. Caller passes the tokenised tail directly; the engine prefills only that tail on top of the pinned seq's stored KV. Returns{error, no_session}for unknown session ids and{error, sticky_busy}for in-flight seqs. Newcache_hit_kind => continuationreports the path inStats. Companion guide section inguides/examples.mdand lifecycle notes ininternals/request-lifecycle.md. ## [0.5.1] - 2026-05-17 Documentation-only patch on top of 0.5.0. ### Added -## Tool-call handlingsection in the README describing what erllama exposes (per-modeltool_call_markers, the{tool_call_delta, }/{erllamatool_call_end, , Full}streaming wire, and the automatic greedy-on-syntax sampler swap) and what it deliberately leaves to the HTTP layer (tool id minting, JSON parsing, canonicalisation). - Newguides/tool-calls.mdcompanion to the README section, linked from the documentation table. - Newinternals/request-lifecycle.mddescribing the per-modelgenstatemadmission, cache resolution, decode, and save pipeline. -internals/c-safety-audit.mdadded to the HexDocs navigation. ### Changed - README rewritten for sharper top-of-funnel: tighter "Why" list, cleaner Quick taste, Common patterns block replacing the old long example. - Architecture diagram corrected —erllama_cache_ramfile_srvanderllama_cache_disk_srvare operator-started standalone servers, not children oferllama_cache_sup. Addederllama_registryanderllama_inflightwhich were missing. - "Inside a request" lifecycle updated for the multi-seq scheduler: two states (idle/running) instead of the v0.1 three-phase model, with co-batchednif_stepand inline thinking/tool-call marker recognition. ## [0.5.0] - 2026-05-16 Tool-call exact-replay scaffolding for downstream HTTP front ends. Models loaded withtool_call_markersproduce structured boundary messages on the streaming wire so a caller can capture the exact bytes the model sampled, store them under a tool id, and splice them back verbatim on later turns to keep the KV-cache prefix match working. Companion primitives expose explicit suffix replay and sticky per-session seq_id pinning. ### Added -tool_call_markers => #{start, end, payload_start (optional), payload_end (optional)}onerllama:load_model/2Config. Each binary is tokenised through the model's own vocabulary at load time; multi-token markers are supported. Omitting the key keeps the backend on the existing path (#39, #42). - Streaming wire oninfer/4gains{erllama_token, Ref, {tool_call_delta, Bin}}per chunk and a single{erllama_tool_call_end, Ref, Full :: binary()}per span, withFullcarrying every emitted delta concatenated so the downstream's exact-replay map stores them verbatim without re-buffering (#39). -erllama:prefill_only/3acceptingOptswithparent_key. When passed a prior turn'sfinish_key, the call warm-restores from that row and prefills only the new suffix before firing the finish save — useful for chaining cache-warming calls across turns (#40). -session_id => term()oninfer/4Params andcomplete/3/prefill_only/3Opts. Pins the underlying seq_id to that session across requests so the next turn whose prompt continues the stored tokens truncates-and-prefills in place on the already-live KV cells (cache_hit_kind => sticky). Concurrent admits on the samesession_idreturn{error, sticky_busy}. Release witherllama:end_session/2(#41). - Per-request greedy sampler swap on tool-call syntax tokens. Models withtool_call_markersbuild a second sampler chain (temperature => 0) at admission; the scheduler routes syntax tokens through it so a tool call is byte-deterministic from a fixed prefix. Optional payload markers flip back to the request's normal sampler for caller-supplied string contents so they stay diverse (#42). ### Changed -step_result()onerllama_model_backendgains four variants ({tool_call_token, },toolcall_end,{tool_call_payload_open, },{toolcall_payload_close, }). Backends without tool-call markers emit none of them. -erllamamodel_stubphase machine re-keyed from sampler ref onto seq_id so mid-request sampler swaps (the new greedy-on-syntax path) don't reset state across ticks.seq_rmnow cleans the per-seq phase entry. ## [0.4.0] - 2026-05-16 Anthropic-Messages compatibility follow-ups: per-request cache delta accounting, a real thinking sampler in the llama.cpp backend, and caller-side thinking-budget clipping. All three are strict additions; existing consumers see no shape change unless they opt in. ### Added -cache_delta => #{read := N, created := N}oncompletion_result(),stats(), andprefill_result()so the downstream Anthropic Messages server can emit accuratecache_creation_input_tokens/cache_read_input_tokensvalues.readis the warm prefix length restored at admission;createdis the largest contribution this request added to the cache beyond that prefix (#35). -thinking_markers => #{start := binary(), end := binary()}onerllama:load_model/2Config. The backend tokenises both strings through the model's vocabulary at load time and thestep/2wrapper maps any sampled token matching a marker into{thinking_token, }orthinkingend. Multi-token markers (BPE-split<think>) are supported. Omitting the key keeps the backend on the non-thinking path (#36). -thinking_signing_keyapplication env. When set, the real backend'sthinking_signature/3HMAC-SHA256s the observed thinking-phase bytes with this key; unset returns<<>>so the downstream omitssignature_delta(#36). -thinking_budget_tokens => pos_integer()oninfer/4Params. Caps the number of{thinking_delta, }payloads delivered before the scheduler synthesises{erllamathinking_end, , }and re-routes further model thinking tokens through the normal post-thinking pipeline (#37). ### Changed -thinking_signature/2callback onerllama_model_backendbumped to/3(third argument is the accumulated thinking bytes). Backwards-compatible: onlyerllama_model_stubimplemented the optional callback in 0.3 and the stub + scheduler + newerllama_model_llamamove in lockstep (#36). ## [0.3.0] - 2026-05-16 Anthropic-Messages compatibility additions on top of 0.2.0: caller-supplied stop sequences with trimmed output, an opt-in extended-thinking message surface with per-block integrity signatures, and a round of NIF safety hardening on the C/C++ side. ### Added -stop_sequences :: [binary()]oninfer/4Paramsandcomplete/3Opts. Generation halts on the first occurrence of any element in the accumulated detokenised output. The match is trimmed from the streamed{erllama_token, , }chunks and the synchronousreply, and the matched binary is reported asstop_sequenceon the result map (complete/3) and stats map (infer/4done message). The key is absent when generation hitlength, was cancelled, or reached EOG without a match. The previously reservedstopplaceholder is renamed tostop_sequences; it was never wired up so this is not a breaking change (#32). -thinking => enabled | disabledoninfer/4Params(defaultdisabled). Whenenabledagainst a thinking-capable backend, streaming requests receive{erllama_token, Ref, {thinking_delta, Bin}}fragments and a single{erllama_thinking_end, Ref, Sig}close marker before any subsequent token.Sigis an opaque integrity signature the downstream forwards verbatim into the Anthropicsignature_deltaSSE event, or<<>>when no signature is available (#33). -erllama_model_backendgains{thinking_token, token_id()}andthinking_endvariants onstep_result()plus an optionalthinking_signature/2callback. Backends without extended thinking emit neither variant and require no changes (#33). ### Changed -llama_batch_init,llama_batch_free, andllama_batch_get_oneare now routed througherllama_safe_batchnoexceptshims so a C++ exception cannot unwind through the C NIF frame (#30). - Per-threadthreadlocalstorage replaces the process-global log buffer used by the malformed-GGUF classifier; concurrent model loads no longer scramble each other'sGGML_ASSERTtext on a NULL return (#31). - NIF unload no longer callsllama_backend_free(avoids apthread_oncewedge on.soreload paths) and clears thellama_log_setcallback so a post-unload log emission cannot dispatch into freed memory (#31). ## [0.2.0] - 2026-05-15 Multi-sequence batched scheduling, map-shaped completion results, chunked prefill, per-model observability, and direct passthrough of the llama.cpp multi-GPU / flash-attention / KV-quant params. ### Changed (breaking) -erllama:complete/2,3anderllama_model:complete/2,3now return{ok, completion_result()}instead of the legacy{ok, ReplyBinary, GeneratedTokens}tuple. The map carries: -reply :: binary()-generated :: [token_id()]-context_tokens :: [token_id()](prompt ++ generated) -committed_tokens :: non_neg_integer()(length(context_tokens)) -finish_key :: cache_key() | undefined— token-exact key for the full context, suitable asparent_keyon the next turn;undefinedif the finish save was suppressed -cache_hit_kind :: exact | partial | cold-finish_reason :: stop | length | cancelled-stats :: stats()Mechanical migration: ```erlang %% before {ok, Reply, _Tokens} = erllama:complete(Model, Prompt). %% after {ok, #{reply := Reply, finish_key := FK}} = erllama:complete(Model, Prompt). ``` - Streaming{erllama_done, Ref, Stats}(infer/4)Statsmap gains two additive keys:finish_keyandcommitted_tokens. No shape break for existing consumers (#21). ### Added #### Multi-sequence batched scheduler (#24, #25, #26, #27) -erllama_nif:step(arity 2) is the new multi-sequence batched decode primitive. Onellama_decodecall mixes prefill and decode rows freely (SARATHI-style co-batching), bounded by the live context'sn_batch. Returns{error, batch_overflow}cleanly so a budget-aware scheduler can shrink and retry, and{error, no_logits}when a decode row has no prefill yet on its seq. - Per-contextper_seq[]tracking (last_logits_idx,next_pos) withERLLAMA_N_SEQ_MAX_CAP = 256.kv_unpack/kv_seq_rmrefresh the per-seq position so subsequentstepcalls see correct state. -erllama_model_backendbehaviour gains optional callbacksstep/2,sampler_new/2,sampler_free/1,seq_rm/2,seq_rm_last/3, plus seq-awarekv_pack/3andkv_unpack/3. All optional; existing backends keep compiling. - The model gen_statem now runs a multi-tenant scheduler. Withcontext_opts.n_seq_max => 1(the default), behaviour is bit-identical to 0.1: exactly one request runs at a time. Settingn_seq_max > 1lets up to N requests prefill and decode concurrently through onellama_decodeper tick. State collapses fromidle/prefilling/generatingtoidle/running; admissions past the seq-id capacity queue FIFO inpending. - Each in-flight request owns its own sampler chain (built at admission viabackend:sampler_new/2, freed at finish), so concurrent requests with differenttemperature/seed/grammarsettings never share sampler state. - Cache save reasons (cold,continued,finish,evict,shutdown) all thread through the request'sseq_idand remain token-exact per-sequence. #### Chunked prefill (#28) -prefill_chunk_sizepolicy knob caps how many tokens a single prefill row contributes to one tick. Defaultmax(64, n_batch div 4); passinfinityto disable. A long prompt is sliced across multiple ticks so it never monopolises the batch and concurrent decoders keep making progress between chunks. Layered on top of then_batchper-tick budget. ####prefill_only/2(#21) -erllama:prefill_only/2anderllama_model:prefill_only/2decode a prompt into KV state and fire a finish save without sampling any output tokens. Returns aprefill_result()map carryingcontext_tokens,committed_tokens,finish_key, andcache_hit_kind. Useful for priming the cache before a burst of short follow-ups, or for holding a warm session across long pauses without consuming generation budget. #### Per-model observability (#22) - New public ETS tableerllama_model_obs, owned byerllama_inflight, written by each model gen_statem on every state transition and read lock-free from any process (including remote nodes viaerpc). - Four new accessors onerllama: -phase/1:idle | prefilling | generatingfor one model id. -pending_len/1: gen_statem pending FIFO depth (calls queued behind whatever is currently running). -last_cache_hit/1:#{kind, prefix_len}of the most recent admission, orundefinedif the model has never admitted. -queue_depth/1: per-model variant of the existing globalqueue_depth/0; counts admitted streaminginfer/4rows. -model_info/1map gainsphase,pending_len, andlast_cache_hitkeys. Additive: existing keys preserved. #### llama.cpp option passthrough (#23) -erllama_nif:load_model(arity 2) now reads three additional keys frommodel_opts: -split_mode :: none | layer | row: multi-GPU split policy. -main_gpu :: non_neg_integer(): GPU index whensplit_mode = none. -tensor_split :: [float()]: per-device proportions (up to 16 entries; shorter lists zero-fill). -erllama_nif:new_context(arity 2) reads three more fromcontext_opts: -flash_attn :: boolean() | auto: enable, disable, or defer to llama.cpp. -type_k,type_v :: f16 | f32 | bf16 | q4_0 | q5_0 | q5_1 | q8_0: KV cache element type for keys and values. - Bad atoms raisebadargbefore the load runs. ### Fixed -warm_restore_primerpassed1instead of the current cell count to the prefill primer, so warm restores that ran the primer at a non-zero offset wrote KV at the wrong position. The primer now takes the live cell count from the per-seq tracker. - The cold prefill path could fire the cold save inside the remainder prefill rather than between the trim prefix and the remainder, leading to a save row that did not match the trim-aligned boundary. The cold save now fires at the cursor-emptied transition between the trim and the remainder. ### Internal - New types exported fromerllama_model:completion_result/0,prefill_result/0. -erllama_model_tnow owns thetensor_splitbuffer; the vendored llama.cpp aliases the pointer rather than copying it, so its storage must outlive the model. -erllama_model_stubderives per-seq tokens fromphash2({decode_step_stub, SeqId, Sampler})so a scheduler bug that swaps samplers between seqs becomes observable in tests. ## [0.1.2] - 2026-05-12 Cluster-routing primitives, speculative-decoding verifier, a cold-path correctness fix, and C-safety CI tooling. All additions are backwards-compatible; existing API call sites unchanged. ### Added #### Cluster routing and load-balancing (#13) -erllama:queue_depth/0returns O(1) inflight count via an atomics counter parked in persistent_term, readable cross-node viaerpc. Used by the upcomingerllama_clusterload balancer (least_loaded, power_of_two strategies). -list_cached_prefixes/2returns the longest cached prefix length of a token list for a given model on this node, across all cache tiers. Used by the cluster cache-affinity router. -erllama_nif:vram_info(arity 0) walks every loaded ggml backend and sums free + total memory across non-CPU devices; returns{error, no_gpu}on a CPU-only build. Used by the cluster scheduler for bin-packing model placement. -erllama:list_models/0map gainsmodel_id,quant_tag,loaded_at_monotonic, andvram_estimate_bkeys. Existing keys (id,pid,status, etc.) are unchanged. #### Speculative decoding (#13) -erllama:draft_tokens/3synchronously generates up toMaxnext-token ids for a prefix. Times out at 30 s with a clean cancel + drain so the caller's mailbox stays clean. Empty prefix is rejected as{error, empty_prefix}. -verify/4runsPrefixTokens ++ Candidatesthrough the model in one forward pass and returns the longest accepted prefix length plus the verifier's own next token. Acceptance walksArgmax[P + i - 1] == c_iand stops at the first mismatch. End-of-generation tokens map to the atomeos. Snapshot + restore protocol leaves the caller's pre-call context view unchanged. Allowed only from the model gen_statem's idle state; non-idle callers receive{error, busy}. - Token-id streaming:erllama_model:stream_emitnow also sends{erllama_token_id, Ref, Id}on every produced token, in addition to the existing{erllama_token, Ref, Bin}. Empty-text tokens (special tokens, BPE merges with no visible bytes) still produce an id message. Existing consumers ignore the new tag. #### Backend behaviour - Optional callbacksextra_metadata/1(vram-related model metadata) andverify/4(speculative verifier) onerllama_model_backend. Backends that omit either get a graceful{error, not_supported}fallback (#13). - Optional callbackseq_clear/1onerllama_model_backend. Llama backend implements it asllama_state_seq_rm(0, 0, -1). Called by the model layer at the top ofenter_prefilling; see the Fixed section below (#16). #### NIFs (#13) -nif_model_size/1,nif_model_n_layer/1,nif_forward_with_argmax/2,nif_vram_info/0. Previously unreachable from Erlang. ### Fixed - Cold-path prefill KV-state leak:erllama_model:enter_prefillingdid not reset the llama_context's KV cache before the new prefill.llama_batch_get_oneauto-positions atn_past, so a second cold request on the same model wrote its prompt KV at[previous_n_past..]instead of[0..], producing different output for the same prompt + seed across calls. The newseq_clear/1callback wipes seq 0 before the cold prefill. Warm restores viakv_unpackwere already correct (#16). ### Changed - Vendoredc_src/llama.cpp/bumped fromb9093tob9119(16 files, mostly Metal/CUDA tweaks plus a newggml-cuda/allreducekernel pair). Publicllama.hAPI used by the NIF is unchanged (#15). -erllama_inflight:register/2andunregister/1switched toets:insert_new/ets:takeso the new atomics counter sees only true admissions and true removals; double-register or double-unregister become observable no-ops (#13). ### Internal - New CI jobs:sanitizers(ASan+UBSan againsttinyllamas/stories260K.ggufunderLD_PRELOAD'd libasan),clang-tidy(NIF sources only), andscan-build(Clang Static Analyzer with--status-bugs) (#14). - Newc_src/CMakeLists.txtoptions:ENABLE_ASAN,ENABLE_TSAN,ENABLE_UBSAN,ENABLE_CLANG_TIDY, scoped to theerllama_niftarget; default OFF. -.clang-tidyconfig at repo root. Vendored llama.cpp is not linted. ### ROADMAP - Pipeline parallelism deferred; blocked on upstream llama.cpp adding layer-range execution tollama_decode. The cluster degrades gracefully viafunction_exported. - Verify context isolation: the current snapshot/restore protocol does not preserve the caller's pre-calldecode_readyflag; callers are assumed to issuedecode_oneimminently. A v2 extension to coverdecode_readyis documented for the next contributor. ## [0.1.1] - 2026-05-12 NIF safety and SIGSEGV hardening. No public API additions; new error tuples surface paths that previously crashed the BEAM or raisedbadargacross a dirty scheduler. ### Fixed - Adapter use-after-free / double-free whenfree_model/1ran while adapter wrappers still referenced the model. The model resource now tracksactive_adaptersalongsideactive_contextsand defersllama_model_freeuntil both reach zero (#10). - Race inset_adapters/2where a concurrentadapter_freecould null the underlying pointer between the per-adapter mutex release and thellama_set_adapters_loracall. Locks are now held across the llama call in pointer-sorted order to defeat AB-BA between concurrent callers (#10). - Per-message memory leak inapply_chat_templatewhen the message list was malformed or allocation failed mid-build. The helper now releases its own role/content allocations on every error path (#10). -prefill/2andembed/2walked past the KV slab when the prompt size reachedn_ctx, and produced undefined behaviour when it exceededn_batch. Both now bounds-check against the live context before touching state, returning{error, context_overflow}or{error, batch_overflow}(#11). -apply_chat_template/2raisedbadargacross the dirty scheduler whencontentwas a list-of-maps (Anthropic-style content blocks) instead of a binary. Returns{error, invalid_content}(#11). -load_model/2surfaced a generic{error, load_failed}for malformed GGUF files. Now returns{error, malformed_gguf}on a best-effort basis when the captured llama log line containsGGML_ASSERT. Best-effort only:llama_log_setis process global and concurrent loads can mis-attribute classification. AGGML_ASSERTthat hitsabort()still terminates the BEAM process; subprocess isolation would be required for a complete fix and is intentionally out of scope (#11). ### Added - New error atoms returned by the NIF:context_overflow,batch_overflow,invalid_content,malformed_gguf. Callers matching{error, }are unaffected; callers that care about the specific reason should match the new atoms. ## [0.1.0] - 2026-05-11 Initial public release. ### Added #### Public API - Native Erlang/OTP wrapper around llama.cpp via a single dirty-scheduler NIF (erllamanif) covering model load, context construction, tokenisation, prefill, single-token decode, and KV pack/unpack. - Models are identified bybinary()on the public API.erllama:load_model/2,complete/2,3,unload/1,status/1,evict/1,shutdown/1takebinary() | pid(). Internal registration uses{via, erllama_registry, BinaryId}so user- supplied ids cannot exhaust the atom table. -erllama:list_models/0returning[model_info()]anderllama:model_info/1keyed on a model id. - Publicerllama:tokenize/2anderllama:detokenize/2keyed on a model id. The low-levelerllama_nif:tokenize(arity 3) anderllama_nif:detokenize(arity 2) remain available. -unload_model/1as an alias forerllama:unload/1matching the OpenAI/Ollama-style naming downstream HTTP servers use. -infer/4streaming inference. Returns{ok, Ref}; tokens are delivered to the caller as{erllama_token, Ref, },{erllamadone, Ref, Stats},{erllama_error, Ref, Reason}. -erllama:cancel/1. Idempotent and fire-and-forget; observed between tokens. -apply_chat_template/2. Renders a normalised chat request (messages,system,tools) through the model's GGUF chat template and tokenises. Backed byllama_chat_apply_template. -erllama:embed/2. Per-sequence pooled embedding viallama_get_embeddings_seqwith last-token fallback. #### Sampling - Sampler parameters:complete/3andinfer/4honourtemperature,top_k,top_p,min_p,repetition_penalty,seed, andgrammarvia one combined chain builder (erllama_nif:configure_sampler(arity 2)). Chain order:grammar -> repetition_penalty -> top_k -> top_p -> min_p -> (temperature > 0 ? temp -> dist(seed) : greedy).set_grammar/2retained as a backwards-compatible alias. - Grammar-constrained sampling: passgrammar => GBNFin thecomplete/3Opts orinfer/4Params; the per-model sampler chain is rebuilt as grammar then greedy for the duration of the request and reset on completion or cancellation. #### LoRA adapters -erllama:load_adapter/2,unload_adapter/2,set_adapter_scale/3,list_adapters/1. Per-adapter sha256 + scale fold into the cache viaerllama_cache_key:effective_fingerprint/2, so rows produced with adapter A never collide with rows from adapter B. Snapshot-at-admission semantics keep in-flight requests on their original fingerprint even if an adapter mutation arrives mid-generation. #### Concurrency model - Concurrent request queue: a secondcomplete/3orinfer/4arriving while one is in flight is queued FIFO instead of getting{error, busy}. The reply{ok, Ref}is sent as soon as the call is admitted; streaming events follow once the queue head advances to the request. - Decode loop schedules each step viagen_statem:cast(self(), decode_step)instead ofnext_eventso cancel, evict, status, and queued requests interleave fairly between tokens. - Seq-aware NIFs (infrastructure for 0.2 multi-seq batching):nif_kv_pack/4accepts an explicitseq_id; newerllama_sampler_tresource owning a standalonellama_sampler*;nif_sampler_new/2,nif_sampler_free/1. Cache rows stay seq-id-free:seq_idis a save/load call argument, never row metadata. #### Internals -erllama_registrymodule: ETS-backedviacallback for binary model ids. -erllama_inflightmodule:Ref -> ModelPidtable socancel/1routes to the right gen_statem. -erllama_model_backendoptional callbacks:apply_chat_template/2,embed/2,set_grammar/2,configure_sampler/2,clear_sampler/1,load_adapter/2,unload_adapter/2,apply_adapters/2. Backends that omit them surface{error, not_supported}from the public API. #### Cache subsystem - Token-exact KV cache with three independently-supervised tiers: RAM (ETS slabs),ram_file(/dev/shm), and disk (plain read I/O). - Sole-writer arbitration througherllama_cache_meta_srv; reads on the hot path go to ETS directly.lookup_exact/1is a single atomicets:lookup(no two-call race) and the meta server cancels waiter timers when an early reply lands so the mailbox doesn't bloat under load. - The disk tier reads files via plainfile:read_file/1into a fresh BEAM heap binary; mmap is deliberately not used. The process already mmaps multi-GB GGUF weights, and a region binary that outlived its closing NIF call would have exposed the BEAM to SIGBUS from any external truncation. - Crash-safe save publish protocol: reserve, write_tmp, check,link(2), mark_published; two-stage TTL cleanup with orphan adoption. - Five save reasons (cold,continued,finish,evict,shutdown) with async/sync semantics matching their use. -saves_droppedcounter: bumps whenever a back-pressured writer pool refuses a save the model wanted to fire. - Multi-turn warmth via explicitparent_keyresume and stateless longest-prefix walk for OpenAI/Anthropic-shaped clients. -erllama_schedulermemory-pressure poller with pluggable sources (memsup,nvidia-smi, custom callback). Off by default. Sweep timer is cancelled onterminate/2so a supervisor restart never leaves a zombie firing into a fresh server. -erllama_cache_writerdirty-IO writer pool with a leak-proof reservation semaphore.pin_and_load/2wraps load + unpack intry/afterso the holder is always checked in. - Persisted hit counters (u32 in disk header) so popular prefixes survive an LRU walk after restart. - End-to-end metrics: hits/misses/saves/evictions plus per-path latency totals (pack_total_ns,load_total_ns,longest_prefix_ns,longest_prefix_probes). #### NIF safety - Per-resourcepthread_mutexand two-resource lifetime pattern for safe concurrentfree/1plus dirty NIF ops. -extern "C" noexceptshim catching every llama.cpp C++ exception at the boundary;decode_onedefensive guard againstGGML_ASSERTaborts. -llama_backend_initis deferred to the firstnif_load_modelviapthread_once, so cache-only and unit-test workloads do not pay theggml_backend_load_allcost at NIF load. -nif_tokenizeandnif_detokenizehonourrelease_pending, so a model returned byfree_model/1as{ok, deferred}cannot be reused via tokenize. -nif_detokenizefails closed onn_vocab <= 0(matchesnif_prefill). -make_errno_atommaps FreeBSD'sEINTEGRITYtoeintegrity. #### Tooling -FindErlang.cmake(adopted from erlang-rocksdb) detectsERTS_INCLUDE_DIRvia the standard CMake find-module contract. - Bench harness (bench/run.sh) with cold-vs-warm matrix and a 4-agent shared-prefix scenario. TinyLlama and LLaMA-3 8B presets. ### CI -actions/checkoutandactions/cachebumped to@v5(Node.js 24). -xref,dialyzer,erlfmt,elvispromoted to gate jobs;build,eunit,proper,ct,freebsddepend on them. - macOS matrix ismacos-14, macos-15. - FreeBSD matrix added:release: ['14.2', '14.4']. Inside the VM: refreshpcre2so git can run, installgit, setgit config --global --add safe.directory ''so llama.cpp's build-infogit rev-parsesucceeds. -erllama_nif_tests:load_model_rejects_non_existent_path_testis now a generator with a 60 s timeout to absorb the lazy Metal init on macOS. ### Tests - 211 EUnit + PropEr property tests + 7 stub Common Test cases. Real-model Common Test suite gated onLLAMA_TEST_MODEL(14 cases including seed determinism, grammar+sampler, apply_chat_template, embeddings, KV pack/unpack round-trip). - New stub-backed coverage: sampler params (erllama_sampler_tests), LoRA adapters + cache identity (erllama_lora_tests), FIFO queueing of concurrent infers (erllama_streaming_tests`). - Multi-platform CI: Ubuntu 24.04 amd64, Ubuntu 24.04 arm64, macOS 14 + 15 (Apple Silicon), FreeBSD 14.2 + 14.4. OTP 28 across the matrix. ### Documentation - README rewritten as a friendly entry point with snippets. - User guides: loading, caching, configuration, building, examples. - Internal design notes: cache design, publish protocol, NIF safety. - ex_doc-friendly module documentation throughout. - ROADMAP.md: what 0.1 does not do yet (multi-seq concurrent decoding, speculative decoding, vision, audio, ONNX/safetensors, stop-sequences, telemetry hooks, multi-GPU pressure, KV compression, cluster). - README closes with a teaser for the upcoming erllama_cluster application: a separate OTP project that coordinates a fleet of erllama nodes (request distribution, cross-node speculative decoding, pipeline parallelism over QUIC). ### Acknowledgements Same idea as antirez/ds4. [Unreleased]: https://github.com/benoitc/erllama/compare/v0.10.1...HEAD [0.10.1]: https://github.com/benoitc/erllama/compare/v0.10.0...v0.10.1 [0.10.0]: https://github.com/benoitc/erllama/compare/v0.9.0...v0.10.0 [0.9.0]: https://github.com/benoitc/erllama/compare/v0.8.0...v0.9.0 [0.8.0]: https://github.com/benoitc/erllama/compare/v0.7.0...v0.8.0 [0.7.0]: https://github.com/benoitc/erllama/compare/v0.6.2...v0.7.0 [0.6.2]: https://github.com/benoitc/erllama/compare/v0.6.1...v0.6.2 [0.6.1]: https://github.com/benoitc/erllama/compare/v0.6.0...v0.6.1 [0.6.0]: https://github.com/benoitc/erllama/compare/v0.5.1...v0.6.0 [0.5.1]: https://github.com/benoitc/erllama/compare/v0.5.0...v0.5.1 [0.5.0]: https://github.com/benoitc/erllama/compare/v0.4.0...v0.5.0 [0.4.0]: https://github.com/benoitc/erllama/compare/v0.3.0...v0.4.0 [0.3.0]: https://github.com/benoitc/erllama/compare/v0.2.0...v0.3.0 [0.2.0]: https://github.com/benoitc/erllama/compare/v0.1.2...v0.2.0 [0.1.2]: https://github.com/benoitc/erllama/compare/v0.1.1...v0.1.2 [0.1.1]: https://github.com/benoitc/erllama/compare/v0.1.0...v0.1.1 [0.1.0]: https://github.com/benoitc/erllama/releases/tag/v0.1.0