Ragex.Search.Keywords (Ragex v0.15.0)

View Source

Extracts weighted keywords from code entities for boosted semantic search.

Keywords are extracted from multiple sources with differential boosting:

  • Documentation (1.5x) -- highest signal, written by humans for humans.
  • Function/module names (1.0x) -- camelCase/snake_case split into tokens.
  • Type specs (0.9x) -- parameter and return types.
  • String literals (0.8x) -- SQL, error messages, domain terms.
  • Comments (0.6x) -- lowest signal, often informal.

Usage

func_info = %{name: :create_user, module: MyApp.Accounts, arity: 2,
               doc: "Creates a new user", metadata: %{
                 strings: ["INSERT INTO users"],
                 comments: ["TODO: add validation"]}}
keywords = Keywords.extract(func_info)
# => %{"create" => 1.0, "user" => 1.5, "INSERT" => 0.8, ...}

Keywords are stored in the graph as function metadata and used by VectorStore during hybrid search for result boosting.

Summary

Types

keyword_map()

Functions

extract(func_info)

Extract weighted keywords from a function info map.

extract_module(module_info)

Extract keywords from a module info map.

relevance_boost(keywords, query_terms)

Compute a relevance boost for a search result based on keyword overlap.

tokenize_name(name)

Split a name (atom or string) into lowercase keyword tokens.

tokenize_text(text)

Tokenize free-form text (docs, comments, strings) into keywords.

Types

keyword_map()

@type keyword_map() :: %{required(String.t()) => float()}

Functions

extract(func_info)

@spec extract(map()) :: keyword_map()

Extract weighted keywords from a function info map.

Parameters

  • func_info -- map with keys :name, :module, :doc, :spec, and :metadata (containing optional :strings and :comments lists).

Returns

%{keyword => weight} where weight reflects the source boost.

extract_module(module_info)

@spec extract_module(map()) :: keyword_map()

Extract keywords from a module info map.

Simpler than function extraction: uses module name and documentation.

relevance_boost(keywords, query_terms)

@spec relevance_boost(keyword_map(), [String.t()]) :: float()

Compute a relevance boost for a search result based on keyword overlap.

Returns a float multiplier (1.0 = no boost, higher = more relevant).

tokenize_name(name)

@spec tokenize_name(atom() | String.t()) :: [String.t()]

Split a name (atom or string) into lowercase keyword tokens.

Handles snake_case, CamelCase, and dot-separated module names.

Examples

iex> Keywords.tokenize_name(:create_user)
["create", "user"]

iex> Keywords.tokenize_name(MyApp.UserAccounts)
["my", "app", "user", "accounts"]

tokenize_text(text)

@spec tokenize_text(String.t()) :: [String.t()]

Tokenize free-form text (docs, comments, strings) into keywords.

Strips punctuation and filters stop words.