Ragex.
View Source
Splits large source texts into overlapping windows for fine-grained embedding.
A single embedding for a 500-line module averages over too much semantic territory. This module produces multiple chunk records per entity, each with its own embedding, all pointing back to the parent entity.
Chunk record format
Each chunk is stored in the embedding table under node type :chunk with a
compound key {parent_type, parent_id, chunk_index}, and text that retains
the parent's identifier header for context anchoring.
Strategy
- Splits on line boundaries to avoid breaking mid-token.
- Uses a sliding window with overlap so context around split points is preserved in adjacent chunks.
- Entities shorter than
min_linesare stored as a single embedding using the normal (non-chunk) path.
Summary
Functions
Build a chunk key suitable for Store.store_embedding/4.
True when the node key looks like a chunk key (3-tuple with integer last element).
Default chunking options as a keyword list. Useful for tests and documentation.
Return the parent {type, id} for a chunk key, or nil for non-chunk keys.
Split text into overlapping line-based windows.
Types
@type chunk_key() :: {parent_type(), parent_id(), non_neg_integer()}
@type parent_id() :: term()
@type parent_type() :: :module | :function | atom()
Functions
@spec chunk_key(parent_type(), parent_id(), non_neg_integer()) :: chunk_key()
Build a chunk key suitable for Store.store_embedding/4.
The key encodes the parent identity and chunk index so all chunks for an entity can be found by scanning the embeddings table.
@spec chunk_key?({parent_type(), parent_id(), non_neg_integer()} | term()) :: boolean()
True when the node key looks like a chunk key (3-tuple with integer last element).
@spec defaults() :: keyword()
Default chunking options as a keyword list. Useful for tests and documentation.
@spec parent_of(chunk_key()) :: {parent_type(), parent_id()} | nil
Return the parent {type, id} for a chunk key, or nil for non-chunk keys.
@spec split( String.t(), keyword() ) :: [{non_neg_integer(), String.t()}]
Split text into overlapping line-based windows.
Returns a list of {chunk_index, chunk_text} pairs. Always returns at
least one chunk. Returns a single-element list when the text is shorter
than min_lines — callers can detect this and skip chunk storage.
Options
:chunk_lines- lines per chunk window (default: 60):overlap_lines- lines shared between adjacent windows (default: 15):min_lines- minimum lines before chunking is applied (default: 30):header- prepended to every chunk for context anchoring (default: "")