Arcana.Graph (Arcana v3.0.1)

Copy Markdown View Source

GraphRAG (Graph-enhanced Retrieval Augmented Generation) for Arcana.

This module provides the public API for GraphRAG functionality:

  • Building knowledge graphs from documents
  • Graph-based search and retrieval
  • Fusion search combining vector and graph results
  • Community summaries for global context

Installation

GraphRAG is optional and requires separate installation:

$ mix arcana.graph.install
$ mix ecto.migrate

Add the NER serving to your supervision tree:

children = [
  MyApp.Repo,
  Arcana.Embedder.Local,
  Arcana.Graph.NERServing  # For entity extraction
]

Configuration

GraphRAG is disabled by default. Enable it in your config:

config :arcana,
  graph: [
    enabled: true,

    # Community detection
    community_levels: 1,      # Hierarchy depth for Leiden algorithm (1 = flat)
    resolution: 1.0,          # Leiden granularity (lower = fewer, larger communities)
    min_size: 1,              # Minimum community size

    # RRF fusion (combining vector + graph search results)
    rrf_k: 60,               # Ranking constant (higher = less weight to top ranks)
    rrf_pool_multiplier: 2,  # Over-fetch multiplier before RRF combine

    # Query-time graph traversal
    query_depth: 0,          # Hops to expand matched entities (0 = direct mentions only)
    query_depth_decay: 0.5,  # Score decay per hop for chunks reached via traversal

    # Community summaries in ask pipeline
    community_summary_limit: 5,  # Max summaries injected as background context
    community_summary_level: 0,  # Level(s) to pull summaries from: integer, list, range or :all

    # Community summarization prompt limits
    summary_max_entities: 50,       # Top N entities by connection count per summary
    summary_max_relationships: 100, # Top N relationships per summary

    # Entity matcher (pluggable)
    entity_matcher: :embedding,      # :embedding (default), :ner, or custom module
    entity_embedding_threshold: 0.3, # Threshold for the :embedding matcher

    # Structured context in ask pipeline (GraphRAG Local Search)
    context_entity_limit: 10,        # Max entity descriptions in LLM context
    context_relationship_limit: 20   # Max relationships in LLM context
  ]

Or enable per-call:

Arcana.ingest(text, repo: MyApp.Repo, graph: true)
Arcana.search(query, repo: MyApp.Repo, graph: true)

Usage

# Build a graph from chunks
{:ok, graph_data} = Arcana.Graph.build(chunks,
  entity_extractor: &MyApp.extract_entities/2,
  relationship_extractor: &MyApp.extract_relationships/3
)

# Convert to queryable format
graph = Arcana.Graph.to_query_graph(graph_data, chunks)

# Search the graph
results = Arcana.Graph.search(graph, entities, depth: 2)

# Fusion search combining vector and graph
results = Arcana.Graph.fusion_search(graph, entities, vector_results)

Components

GraphRAG consists of several modules:

Custom Implementations

All core extractors and detectors support the behaviour pattern for extensibility:

# Custom entity extractor
config :arcana, :graph,
  entity_extractor: {MyApp.SpacyExtractor, endpoint: "http://localhost:5000"}

# Custom relationship extractor
config :arcana, :graph,
  relationship_extractor: {MyApp.PatternExtractor, patterns: [...]}

# Custom community detector
config :arcana, :graph,
  community_detector: {MyApp.LouvainDetector, resolution: 0.5}

# Custom community summarizer
config :arcana, :graph,
  community_summarizer: {MyApp.ExtractiveSum, max_sentences: 3}

Summary

Functions

build(chunks, opts)

Builds graph data from document chunks.

build_and_persist(chunk_records, collection, repo, opts)

Builds and persists graph data from chunk records during ingest.

community_summaries(graph, opts \\ [])

Gets community summaries from the graph.

config()

Returns the current GraphRAG configuration.

enabled?()

Returns whether GraphRAG is enabled globally.

expand_entity_ids(entity_ids, depth, collection_ids, opts)

Expands entity ids through the relationships table for depth hops.

find_entities(graph, name, opts \\ [])

Finds entities in the graph by name.

fusion_search(graph, entities, vector_results, opts \\ [])

Combines vector search and graph search using Reciprocal Rank Fusion.

installed?(repo, opts \\ [])

Returns whether the GraphRAG schema is installed in the database.

normalize_levels(level)

Normalizes a level selection into :all or a list of levels.

query_depth(opts)

Resolves the query-time graph traversal depth from per-call opts and global config.

resolve_entity_extractor(opts)

Resolves the entity extractor from options and config.

search(graph, entities, opts \\ [])

Searches the knowledge graph for relevant chunks.

summary_levels(config \\ config())

Returns the community hierarchy levels that queries read summaries from.

to_query_graph(graph_data, chunks)

Converts builder output to queryable graph format.

traverse(graph, entity_id, opts \\ [])

Traverses the graph from a starting entity.

Functions

build(chunks, opts)

Builds graph data from document chunks.

Delegates to Arcana.Graph.GraphBuilder.build/2.

Options

  • :entity_extractor - Function to extract entities from text
  • :relationship_extractor - Function to extract relationships

Example

{:ok, graph_data} = Arcana.Graph.build(chunks,
  entity_extractor: fn text, _opts ->
    Arcana.Graph.EntityExtractor.NER.extract(text, [])
  end,
  relationship_extractor: fn text, entities, _opts ->
    Arcana.Graph.RelationshipExtractor.extract(text, entities, my_llm)
  end
)

build_and_persist(chunk_records, collection, repo, opts)

Builds and persists graph data from chunk records during ingest.

Processes chunks incrementally, persisting after each chunk so progress is saved continuously. Accepts an optional :progress callback that receives {current_chunk, total_chunks} after each chunk is processed.

Options

  • :progress - Callback function fn current, total -> ... end called after each chunk

Examples

# With progress logging
Arcana.Graph.build_and_persist(chunks, collection, repo,
  progress: fn current, total ->
    IO.puts("Processed chunk #{current}/#{total}")
  end
)

community_summaries(graph, opts \\ [])

Gets community summaries from the graph.

Community summaries provide high-level context about clusters of related entities, useful for global queries.

Options

  • :level - Filter by hierarchy level (0 = finest)
  • :entity_id - Filter by communities containing entity

Example

# Get all top-level summaries
summaries = Arcana.Graph.community_summaries(graph, level: 0)

config()

Returns the current GraphRAG configuration.

Example

Arcana.Graph.config()
# => %{enabled: false, community_levels: 1, resolution: 1.0}

enabled?()

Returns whether GraphRAG is enabled globally.

Check this before performing graph operations:

if Arcana.Graph.enabled?() do
  # Build graph during ingest
end

expand_entity_ids(entity_ids, depth, collection_ids, opts)

Expands entity ids through the relationships table for depth hops.

Walks arcana_graph_relationships breadth-first (both directions) from the given entity ids, returning a map of hop distance to the entity ids first discovered at that hop: %{0 => direct_ids, 1 => neighbors, ...}. Each entity appears once, at its minimal hop distance.

collection_ids follows the usual scoping semantics: nil is unscoped, a list restricts discovered neighbors to those collections, and an empty list matches nothing (no neighbors are pulled in).

Options

  • :repo - Ecto repo (required)

find_entities(graph, name, opts \\ [])

Finds entities in the graph by name.

Options

  • :fuzzy - Enable fuzzy matching (default: false)

fusion_search(graph, entities, vector_results, opts \\ [])

Combines vector search and graph search using Reciprocal Rank Fusion.

This is the primary retrieval method for GraphRAG, merging results from both vector similarity and knowledge graph traversal.

Options

  • :depth - Graph traversal depth (default: 1)
  • :limit - Maximum results to return (default: 10)
  • :k - RRF constant (default: 60)

Example

# Run vector search separately
{:ok, vector_results} = Arcana.search(query, repo: MyApp.Repo)

# Extract entities from query
{:ok, entities} = Arcana.Graph.EntityExtractor.NER.extract(query, [])

# Combine with graph search
results = Arcana.Graph.fusion_search(graph, entities, vector_results)

installed?(repo, opts \\ [])

Returns whether the GraphRAG schema is installed in the database.

Separate from enabled?/0, which only reads config. The graph tables ship on their own migration version, so an app can legitimately run Arcana.Migration.up/1 and skip Arcana.Graph.Migration.up/1 - and then a graph query fails with relation "arcana_graph_entities" does not exist rather than returning nothing. Config being on says the operator wants graph features; this says the database can actually answer.

if Arcana.Graph.installed?(MyApp.Repo) do
  # safe to query entities, relationships, communities
end

Checks the table rather than the arcana_graph:<n> version marker on purpose: the marker can be missing on an install that predates versioning, or that had its table comment clobbered, and those databases can still be queried.

Options

  • :prefix - the Postgres schema to look in. Without it the question is whether an unqualified query would find the table, which is what the callers actually do

normalize_levels(level)

Normalizes a level selection into :all or a list of levels.

query_depth(opts)

Resolves the query-time graph traversal depth from per-call opts and global config.

Reads :graph_depth from opts first, then falls back to config :arcana, graph: [query_depth: n] (default 0). Raises ArgumentError on anything other than a non-negative integer.

resolve_entity_extractor(opts)

Resolves the entity extractor from options and config.

search(graph, entities, opts \\ [])

Searches the knowledge graph for relevant chunks.

Finds entities matching the query, traverses relationships, and returns connected chunks.

Options

  • :depth - How many hops to traverse (default: 1)

Example

entities = [%{name: "OpenAI", type: :organization}]
results = Arcana.Graph.search(graph, entities, depth: 2)

summary_levels(config \\ config())

Returns the community hierarchy levels that queries read summaries from.

Configured with community_summary_level, which accepts an integer, a list of integers, a range, or :all. Summarization consumes the same key, so the levels that get summarized and the levels ask/2 reads can't drift apart.

Returns :all or a list of levels.

Examples

Arcana.Graph.summary_levels()
# => [0]

to_query_graph(graph_data, chunks)

Converts builder output to queryable graph format.

Delegates to Arcana.Graph.GraphBuilder.to_query_graph/2.

traverse(graph, entity_id, opts \\ [])

Traverses the graph from a starting entity.

Options

  • :depth - Maximum traversal depth (default: 1)