AshScylla.Search.Storage (AshScylla v1.9.0)

CQL schema definitions for the inverted index tables used by the search engine.

Provides functions to create and drop the required tables for the inverted index. The tables are:

search_post_terms — inverted index mapping (term, shard, post_id) to the term's total frequency within the document (summed across fields)
search_post_fields — per-field term/frequency maps used to compute diff-based updates

Postings are stored at document level: one row per (term, post_id). Queries fetch F× fewer rows than per-field layouts (F = number of fields), which is the dominant cost when reading hot terms.

Usage

AshScylla.Search.Storage.create_tables(MyApp.Repo, "my_keyspace")
AshScylla.Search.Storage.drop_tables(MyApp.Repo, "my_keyspace")

Summary

Functions

create_post_fields_cql(keyspace)

Returns the CQL statement to create the search_post_fields table.

create_post_terms_cql(keyspace)

Returns the CQL statement to create the search_post_terms table.

create_tables(repo, keyspace)

Creates all search engine tables in the given keyspace.

drop_tables(repo, keyspace)

Drops all search engine tables from the given keyspace.

fetch_field_maps(repo, keyspace, post_id)

Fetches every stored field map for a post from search_post_fields.

shard_for(term, num_shards \\ 16)

Computes the shard number for a given term.

Functions

create_post_fields_cql(keyspace)

@spec create_post_fields_cql(String.t()) :: String.t()

Returns the CQL statement to create the search_post_fields table.

Stores the analyzed terms with their frequencies for each post field. Used during updates to diff old vs new content and recompute document-level totals.

create_post_terms_cql(keyspace)

@spec create_post_terms_cql(String.t()) :: String.t()

Returns the CQL statement to create the search_post_terms table.

This is the primary inverted index table. Each row maps a term to a post_id with the term's total frequency in that document.

Partition key: (term, shard) to avoid hotspot partitions for common terms. Clustering key: post_id.

create_tables(repo, keyspace)

@spec create_tables(module(), String.t()) :: :ok | {:error, term()}

Creates all search engine tables in the given keyspace.

Returns :ok on success or {:error, reason} on failure.

drop_tables(repo, keyspace)

@spec drop_tables(module(), String.t()) :: :ok | {:error, term()}

Drops all search engine tables from the given keyspace.

Returns :ok on success or {:error, reason} on failure.

fetch_field_maps(repo, keyspace, post_id)

@spec fetch_field_maps(module(), String.t(), String.t()) ::
  {:ok, %{required(String.t()) => %{required(String.t()) => pos_integer()}}}
  | {:error, term()}

Fetches every stored field map for a post from search_post_fields.

Returns %{"field_name" => %{"term" => tf}}. Empty map if the post has no indexed fields.

shard_for(term, num_shards \\ 16)

@spec shard_for(String.t(), non_neg_integer()) :: non_neg_integer()

Computes the shard number for a given term.

Uses :erlang.phash2/2 to distribute terms across the configured number of shards. This prevents hotspot partitions for high-frequency terms.