Getting Started with Arcana 🔮📚

Arcana is a RAG (Retrieval Augmented Generation) library for Elixir that lets you build AI-powered search and question-answering into your Phoenix applications.

Installation

With Igniter (recommended):

mix igniter.install arcana
mix ecto.migrate

This adds the dependency, creates migrations, configures your repo, and sets up the dashboard route.

The generated migration delegates to Arcana rather than spelling out DDL, so upgrading later is another migration calling up/1 again:

defmodule MyApp.Repo.Migrations.AddArcana do
  use Ecto.Migration

  def up, do: Arcana.Migration.up(dimensions: 384)

  def down, do: Arcana.Migration.down()
end

up/1 requires :dimensions, which mix arcana.install fills in from your configured embedder. It also takes :prefix to install into a Postgres schema of its own. Installs predating versioned migrations converge by running it: version 1 creates only what is absent and adds only what a later release introduced, without dropping anything.

Without Igniter:

def deps do
  [
    {:arcana, "~> 2.0"}
  ]
end

mix deps.get
mix arcana.install
mix ecto.migrate

Embedding Configuration

Arcana uses local embeddings by default via Bumblebee. No API keys needed.

bumblebee is an optional dependency, so add it (plus a backend) when using the default local embedder:

def deps do
  [
    {:arcana, "~> 2.0"},
    {:bumblebee, "~> 0.6"},
    {:exla, "~> 0.10"}
  ]
end

Apps that bring their own embedder (the Arcana.Embedder behaviour) can skip both and avoid compiling the Bumblebee/tokenizers toolchain. The same goes for req_llm: it's optional and only needed when using model strings (like "openai:gpt-4o-mini") as the LLM or Arcana.Loop.

# config/config.exs

# Default - BGE Small (384 dimensions, 133MB)
config :arcana, embedder: :local

# Use a different model
config :arcana, embedder: {:local, model: "BAAI/bge-base-en-v1.5"}

Add Arcana components to your supervision tree:

# application.ex
children = [
  MyApp.Repo,
  ArcanaWeb.TaskSupervisor,  # Required for dashboard async operations
  Arcana.Embedder.Local   # Only if using local embeddings
]

ArcanaWeb.TaskSupervisor is required for the dashboard's async operations (Ask, Maintenance). Arcana.Embedder.Local starts the local embedding model (only needed if using local embeddings).

Available Models

E5 Models: E5 models require special prefixes (query: for search queries, passage: for documents). Arcana handles this automatically - just configure the model and the prefixes are added during search and ingestion.

Changing Embedding Models

When switching to a model with different dimensions, you need to resize the vector column:

# 1. Update your config to use the new model
# 2. Generate a migration to resize the vector column
mix arcana.gen.embedding_migration

# 3. Run the migration
mix ecto.migrate

# 4. Re-embed all documents with the new model
mix arcana.reembed

For OpenAI embeddings or custom providers, see the LLM Integration guide.

Chunking Configuration

Arcana uses the default text chunker which splits documents into overlapping chunks:

# config/config.exs

# Default - 450 tokens with 50 token overlap
config :arcana, chunker: :default

# Custom chunk sizes
config :arcana, chunker: {:default, chunk_size: 512, chunk_overlap: 100}

Available Options

Custom Chunkers

For semantic chunking or domain-specific splitting, implement the Arcana.Chunker behaviour:

defmodule MyApp.SemanticChunker do
  @behaviour Arcana.Chunker

  @impl true
  def chunk(text, opts) do
    # Split on semantic boundaries (paragraphs, sections, etc.)
    text
    |> split_semantically()
    |> Enum.with_index()
    |> Enum.map(fn {text, index} ->
      %{text: text, chunk_index: index, token_count: estimate_tokens(text)}
    end)
  end
end

# Configure globally
config :arcana, chunker: MyApp.SemanticChunker

# Or per-ingest
Arcana.ingest(text, repo: MyApp.Repo, chunker: MyApp.SemanticChunker)

PDF Parsing Configuration

Arcana supports PDF file ingestion with pluggable parsers. The default uses Poppler's pdftotext command-line tool.

Default Parser (Poppler)

# config/config.exs

# Default: Poppler's pdftotext
config :arcana, pdf_parser: :poppler

# With options
config :arcana, pdf_parser: {:poppler, layout: true}

Installing Poppler:

Check availability:

iex> Arcana.FileParser.PDF.Poppler.available?()
true

Poppler Options

Custom PDF Parsers

For alternative PDF parsing (e.g., Apache PDFBox, pdf2htmlex, cloud APIs), implement the Arcana.FileParser.PDF behaviour:

defmodule MyApp.PDFBoxParser do
  @behaviour Arcana.FileParser.PDF

  @impl true
  def parse(path, opts) when is_binary(path) do
    # Your PDF parsing logic
    case extract_with_pdfbox(path, opts) do
      {:ok, text} -> {:ok, text}
      {:error, reason} -> {:error, reason}
    end
  end

  # Optional: declare support for binary content
  # (default is false - parser only accepts file paths)
  def supports_binary?, do: false

  defp extract_with_pdfbox(path, _opts) do
    # Call PDFBox CLI, Rustler NIF, or port
    {:ok, "extracted text"}
  end
end

Configure your custom parser:

# config/config.exs
config :arcana, pdf_parser: MyApp.PDFBoxParser
config :arcana, pdf_parser: {MyApp.PDFBoxParser, some_option: "value"}

Binary Content Support

Some parsers can accept binary PDF content directly (useful for processing uploads without saving to disk). Declare this capability:

defmodule MyApp.InMemoryPDFParser do
  @behaviour Arcana.FileParser.PDF

  @impl true
  def parse(binary, opts) when is_binary(binary) do
    # Parse binary PDF content directly
    {:ok, extracted_text}
  end

  def supports_binary?, do: true
end

Check if a parser supports binary input:

iex> Arcana.FileParser.PDF.supports_binary?({MyApp.InMemoryPDFParser, []})
true

iex> Arcana.FileParser.PDF.supports_binary?({Arcana.FileParser.PDF.Poppler, []})
false

Basic Usage

Ingesting Documents

# Ingest text content
{:ok, document} = Arcana.ingest("Your content here", repo: MyApp.Repo)

# With metadata
{:ok, document} = Arcana.ingest(
  "Article about Elixir",
  repo: MyApp.Repo,
  metadata: %{"author" => "Jane", "category" => "programming"}
)

# With a source ID for grouping
{:ok, document} = Arcana.ingest(
  "Chapter 1 content",
  repo: MyApp.Repo,
  source_id: "book-123"
)

# Ingest from file (supports .txt, .md, .pdf)
{:ok, document} = Arcana.ingest_file("path/to/document.pdf", repo: MyApp.Repo)

# Organize documents into collections
{:ok, document} = Arcana.ingest(
  "Product documentation",
  repo: MyApp.Repo,
  collection: "products"
)

# With collection description (helps Pipeline.select/2 route to the right collection)
{:ok, document} = Arcana.ingest(
  "API reference",
  repo: MyApp.Repo,
  collection: %{name: "api", description: "REST API endpoints and parameters"}
)

Note: PDF support requires a PDF parser. The default uses Poppler's pdftotext. See PDF Parsing Configuration for installation and custom parsers.

Listing and Reading Documents

Build admin surfaces or assert on ingestion outcomes without touching Arcana's schemas directly:

# List a collection's documents (newest first, collection preloaded)
{:ok, docs} = Arcana.list_documents(repo: MyApp.Repo, collection: "products")

# Filter by status or source, paginate
{:ok, failed} = Arcana.list_documents(repo: MyApp.Repo, status: :failed)
{:ok, page2} = Arcana.list_documents(repo: MyApp.Repo, limit: 20, offset: 20)
{:ok, total} = Arcana.count_documents(repo: MyApp.Repo, collection: "products")

# Read one, delete one
{:ok, document} = Arcana.get_document(id, repo: MyApp.Repo)
:ok = Arcana.delete(id, repo: MyApp.Repo)

Arcana.Document is the stable public shape: id, status, chunk_count, metadata, source_id, content_type, timestamps, and the preloaded collection.

Searching

# Vector search (default)
{:ok, results} = Arcana.search("functional programming", repo: MyApp.Repo)

# Keyword search
{:ok, results} = Arcana.search("Elixir", repo: MyApp.Repo, mode: :keyword)

# Hybrid search (combines semantic + fulltext)
{:ok, results} = Arcana.search("Elixir patterns", repo: MyApp.Repo, mode: :hybrid)

# Hybrid with custom weights (pgvector backend)
{:ok, results} = Arcana.search("Elixir patterns",
  repo: MyApp.Repo,
  mode: :hybrid,
  vector_weight: 0.7,  # Weight for semantic similarity
  keyword_weight: 0.3   # Weight for keyword matching
)

# With filters
{:ok, results} = Arcana.search("query",
  repo: MyApp.Repo,
  limit: 5,
  threshold: 0.7,
  source_id: "book-123",
  collection: "products"  # Filter by collection
)

Every mode returns the same Arcana.SearchResult struct, with id, text, document_id, chunk_index, score, per-mode score breakdowns (vector_score/keyword_score, set by single-query hybrid search), and the chunk's stored metadata.

Strict Collection Scoping

By default, searching a collection name that doesn't exist silently searches across all collections, and ingesting into one creates it on the fly. In multi-tenant apps that's dangerous: a typo'd tenant collection becomes a cross-tenant search. Turn on strict mode to make unknown collection names an error instead:

config :arcana, strict_collections: true

With strict mode on, Arcana.search/2, Arcana.ingest/2, vector-store writes/deletes, and maintenance functions return {:error, {:unknown_collection, name}} for names that don't exist — Arcana.ask/2 wraps it as its usual {:error, {:search_failed, {:unknown_collection, name}}}. Create collections explicitly with Arcana.Collection.get_or_create/3. The flag can also be passed per call (strict_collections: false opts out for one call).

Three things to know:

• Ingest without a :collection uses the implicit "default" collection, which strict mode also requires to exist. Create it once at setup: Arcana.Collection.get_or_create("default", MyApp.Repo).
• Without strict mode, an unknown name still searches unscoped on the vector/keyword path (legacy behavior), but graph-enhanced context is scoped to nothing rather than leaking across collections.
• Strict validation needs the repo-backed collection registry. The :memory vector store has no registry: it always fails closed for unknown collections (empty results) instead of returning the error.

Strict mode will become the default in Arcana 3.0.

Question Answering

Use Arcana.ask/2 to combine search with an LLM for answers:

llm_fn = fn prompt, context ->
  # Call your LLM API here
  {:ok, "Generated answer based on context"}
end

{:ok, answer} = Arcana.ask("What is Elixir?",
  repo: MyApp.Repo,
  llm: llm_fn,
  limit: 5
)

See the LLM Integration guide for production-ready LLM integration.

Pipeline (Modular RAG)

For more control over the RAG process, use Arcana.Pipeline. See the Pipeline guide for the full reference; this section just shows the shape.

alias Arcana.Pipeline

llm = fn prompt -> {:ok, "LLM response"} end

ctx =
  Pipeline.new("Compare Elixir and Erlang", repo: MyApp.Repo, llm: llm)
  |> Pipeline.select(collections: ["elixir-docs", "erlang-docs"])
  |> Pipeline.expand()
  |> Pipeline.search()
  |> Pipeline.answer()

ctx.answer

Pipeline Steps

Expand vs. Decompose

Use expand/2 when queries contain abbreviations, jargon, or domain-specific terms:

# Before expand: "ML models"
# After expand: "ML machine learning artificial intelligence models algorithms"

ctx
|> Pipeline.expand()
|> Pipeline.search()

Use decompose/2 when questions have multiple parts:

# Before decompose: "What is X and how does it compare to Y?"
# After decompose: ["What is X?", "How does it compare to Y?"]

ctx
|> Pipeline.decompose()
|> Pipeline.search()  # Searches each sub-question

You can combine both:

ctx
|> Pipeline.expand()      # Adds synonyms to the original question
|> Pipeline.decompose()   # Splits into sub-questions
|> Pipeline.search()      # Searches each expanded sub-question

Self-Correcting Search

Enable automatic query refinement when results are insufficient:

ctx
|> Pipeline.search(self_correct: true, max_iterations: 3)

The pipeline will:

1. Execute the search
2. Ask the LLM if results are sufficient
3. If not, rewrite the query and retry
4. Repeat until sufficient or max iterations reached

Re-ranking

Improve result quality by re-scoring chunks after retrieval:

ctx
|> Pipeline.search()
|> Pipeline.rerank(threshold: 7)  # Keep chunks scoring 7+/10
|> Pipeline.answer()

The LLM scores each chunk's relevance to the question. Chunks below the threshold are filtered out, and remaining chunks are sorted by score.

For custom re-ranking logic (e.g., cross-encoder models):

# Custom reranker module
defmodule MyApp.CrossEncoderReranker do
  @behaviour Arcana.Reranker

  @impl true
  def rerank(question, chunks, _opts) do
    # Your scoring logic here
    {:ok, scored_chunks}
  end
end

ctx |> Pipeline.rerank(reranker: MyApp.CrossEncoderReranker)

Query Rewriting

Improve search results by rewriting queries before searching:

alias Arcana.Rewriters

# Create a rewriter with your LLM
rewriter = Rewriters.expand(llm: fn prompt ->
  # Call LLM to expand the query
  {:ok, "expanded query with synonyms"}
end)

# Use it with search
results = Arcana.search("ML",
  repo: MyApp.Repo,
  rewriter: rewriter
)

Dashboard UI

Arcana includes a LiveView dashboard for managing documents:

# In your router
import ArcanaWeb.Router

scope "/admin", MyAppWeb do
  pipe_through [:browser, :admin]
  arcana_dashboard("/arcana", repo: MyApp.Repo)
end

Telemetry

Arcana emits telemetry events for all operations. You can attach handlers to observe performance and usage:

# In your application startup
:telemetry.attach_many(
  "my-arcana-handler",
  [
    [:arcana, :ingest, :stop],
    [:arcana, :search, :stop],
    [:arcana, :ask, :stop]
  ],
  fn event, measurements, metadata, _config ->
    duration_ms = System.convert_time_unit(measurements.duration, :native, :millisecond)
    Logger.info("#{inspect(event)} completed in #{duration_ms}ms")
  end,
  nil
)

Events follow the :telemetry.span/3 convention with :start, :stop, and :exception suffixes. See Arcana.Telemetry for complete documentation.

Next Steps

• LLM Integration - Connect Arcana to LLMs
• Pipeline (Modular RAG) - Compose retrieval steps yourself
• Loop (Agentic RAG) - Let an LLM drive tool calls until it can answer
• Re-ranking - Improve retrieval quality
• Search Algorithms - Hybrid search modes
• Evaluation - Measure retrieval quality
• Dashboard - Web UI setup