Cherry.Search.Index (cherry v0.4.1)

Copy Markdown View Source

Builds the search index emitted for search: "cherry" — the built-in engine that needs no Node, no npm, and no network (DESIGN.md §7).

The index is an inverted list: every indexed token maps to the documents containing it and a weight. Ranking happens in the browser, so the emitted file carries weights and document metadata, not scores.

%{
  "v" => 1,
  "docs" => [%{"u" => "introsort/", "t" => "Introsort", "e" => "…", "d" => "2014-03-02"}],
  "terms" => %{"introsort" => [[0, 9]]}
}

terms values are [document_index, weight] pairs, pointing into docs by position. Weight is term frequency with the title counted three times and tags twice, so a word in a title outranks the same word buried in a paragraph.

Fenced code blocks are stripped before tokenizing. Prose is what readers search for, and a technical archive's code samples otherwise dominate the index with symbols and language keywords.

Summary

Types

t()

The index as plain data, before JSON encoding.

Functions

build(documents, site)

Builds the index for every routable, non-raw document.

render(documents, site)

Builds the index and encodes it as deterministic JSON.

tokenize(text)

Splits text into index tokens.

Types

t()

@type t() :: %{required(String.t()) => term()}

The index as plain data, before JSON encoding.

Functions

build(documents, site)

@spec build([Cherry.Content.Document.t()], Cherry.Site.t()) :: t()

Builds the index for every routable, non-raw document.

Documents are ordered by output path so the same content always produces the same document indices, which keeps the build byte-stable.

render(documents, site)

@spec render([Cherry.Content.Document.t()], Cherry.Site.t()) :: String.t()

Builds the index and encodes it as deterministic JSON.

tokenize(text)

@spec tokenize(String.t()) :: [String.t()]

Splits text into index tokens.

Public because the browser island must tokenize queries exactly the same way; the two implementations are checked against each other in test/cherry/search_index_test.exs.