Ragex.Analyzers.DeeperIndexing (Ragex v0.18.1)

View Source

Post-processing pass that extracts string literals and comments from source code and associates them with the nearest function in the analysis result.

This enriches the knowledge graph with deeper metadata enabling:

  • Searching for SQL queries, error messages, and domain terms inside code.
  • Associating inline TODO/HACK/FIXME comments with specific functions.
  • Keyword-boosted semantic search via Ragex.Search.Keywords.

Supported Languages

  • Elixir -- walks the AST for binary literals; re-uses line-based comment extraction.
  • Erlang -- walks parsed forms for {string, Line, Chars} tuples; regex for %-style comments.
  • Python -- regex extraction of "..." / '...' / triple-quoted strings and #-style comments.
  • JavaScript/TypeScript -- regex extraction of string literals, template literals, and // / /* */ comments.

Usage

{:ok, analysis} = Ragex.Analyzers.Elixir.analyze(source, path)
enrichment = DeeperIndexing.extract(source, path, analysis)
# => %{strings: %{{mod, func, arity} => ["INSERT INTO ...", ...]},
#      comments: %{{mod, func, arity} => ["TODO: refactor", ...]}}

Summary

Types

enrichment()
func_key()

Functions

extract(source, file_path, analysis)

Extract string literals and comments from source code, associating each with the nearest function from the analysis result.

extract_comments(source, arg2)

Extract comments with line numbers from source code.

extract_strings(source, arg2)

Extract string literals with line numbers from source code.

merge_into_analysis(enrichment, analysis)

Merge enrichment data into function metadata maps.

Types

enrichment()

@type enrichment() :: %{
  strings: %{required(func_key()) => [String.t()]},
  comments: %{required(func_key()) => [String.t()]}
}

func_key()

@type func_key() :: {atom(), atom(), non_neg_integer()} | :module_level

Functions

extract(source, file_path, analysis)

@spec extract(String.t(), String.t(), map()) :: enrichment()

Extract string literals and comments from source code, associating each with the nearest function from the analysis result.

Parameters

  • source -- raw source code string
  • file_path -- path used for language detection
  • analysis -- the %{functions: [...], ...} map from an analyzer

Returns

%{strings: %{func_key => [str, ...]}, comments: %{func_key => [str, ...]}} where func_key is {module, name, arity} or :module_level.

extract_comments(source, arg2)

@spec extract_comments(String.t(), atom()) :: [{pos_integer(), String.t()}]

Extract comments with line numbers from source code.

Returns a list of {line, comment_text} tuples. Merges consecutive comment lines into single blocks.

extract_strings(source, arg2)

@spec extract_strings(String.t(), atom()) :: [{pos_integer(), String.t()}]

Extract string literals with line numbers from source code.

Returns a list of {line, string_content} tuples.

merge_into_analysis(enrichment, analysis)

@spec merge_into_analysis(enrichment(), map()) :: map()

Merge enrichment data into function metadata maps.

Takes the enrichment from extract/3 and the original analysis, returning updated function info maps with :strings, :comments added to metadata.