Arcana (Arcana v4.0.0)

Copy Markdown View Source

RAG (Retrieval Augmented Generation) library for Elixir.

Arcana provides document ingestion, embedding, and vector search capabilities that you can embed into any Phoenix/Ecto application.

Usage

# Ingest a document
{:ok, document} = Arcana.ingest("Your text content", repo: MyApp.Repo)

# Search for relevant chunks
{:ok, results} = Arcana.search("your query", repo: MyApp.Repo)

# Ask questions with RAG
{:ok, answer} = Arcana.ask("What is X?", repo: MyApp.Repo, llm: my_llm)

# Delete a document
:ok = Arcana.delete(document.id, repo: MyApp.Repo)

Modules

Summary

Functions

Asks a question using retrieved context from the knowledge base. See Arcana.Ask.ask/2 for options.

Returns the configured chunker as a {module, opts} tuple. See Arcana.Config for configuration options.

Returns the current Arcana configuration.

Counts documents matching the list_documents/1 filters. See Arcana.Documents.count_documents/1.

Deletes a document and all its chunks.

Returns the configured embedder as a {module, opts} tuple. See Arcana.Config for configuration options.

Fetches sparse metadata for published documents in one explicitly scoped read. See Arcana.Documents.get_document_metadata/2.

Returns whether GraphRAG is enabled.

Ingests text content, creating a document with embedded chunks. See Arcana.Ingest.ingest/2 for options.

Ingests in-memory bytes, routing on the required :filename option's extension. See Arcana.Ingest.ingest_binary/2 for options.

Ingests a file, parsing its content and creating a document with embedded chunks. See Arcana.Ingest.ingest_file/2 for options.

Lists documents, newest first. See Arcana.Documents.list_documents/1.

Rewrites a query using a provided rewriter function. See Arcana.Search.rewrite_query/2 for options.

Searches for chunks similar to the query. See Arcana.Search.search/2 for options.

Functions

ask(question, opts)

Asks a question using retrieved context from the knowledge base. See Arcana.Ask.ask/2 for options.

chunker()

Returns the configured chunker as a {module, opts} tuple. See Arcana.Config for configuration options.

config()

Returns the current Arcana configuration.

count_documents(opts)

Counts documents matching the list_documents/1 filters. See Arcana.Documents.count_documents/1.

delete(document_id, opts)

Deletes a document and all its chunks.

When the graph is enabled, also sweeps the document's collection for orphaned graph data: entities left with zero mentions are deleted and communities that referenced them are marked dirty so the next summarize pass regenerates them.

Returns :ok, {:error, :not_found}, {:error, {:sweep_failed, reason}} when the graph store returns an error from its sweep, or {:error, reason} for a database failure such as a foreign key violation, or a not-null violation from another table pointing at the document.

A sweep that raises a database error reports as a database failure rather than as :sweep_failed — so match {:error, _} as well if you route on the sweep case. sweep_orphans/2 deletes through repo.delete_all/1, which skips changeset constraint mapping, so a host row referencing an entity the sweep wants to remove arrives that way.

When a non-Ecto graph store, including the built-in :memory store, is cleaned after the database delete commits, a returned error or exception is wrapped as {:error, {:post_commit_graph_cleanup_failed, context}}. The context carries :reason, :chunk_ids, :published_chunk_ids, and :collection_id, so the graph deletion and sweep can be retried even though the document row is already gone. Calling delete/2 inside an existing Ecto transaction when an a non-Ecto graph cleanup is required returns {:error, :external_graph_store_requires_post_commit_delete} before deleting.

A concurrent delete reports {:error, :not_found}, the same as losing that race by a moment more would have.

Infrastructure failures — a lost connection, a pool timeout — still raise. There is nothing useful to hand back and the transaction's fate is unknown, so those are left to the caller's supervisor rather than flattened into a tuple that would read like a clean refusal.

Called on its own, the delete and the orphan sweep run in one transaction, so a :sweep_failed leaves the document in place: nothing happened and the call can be retried. That is worth knowing if you are upgrading, because this case used to delete the document and report the failed cleanup afterwards. Inside a transaction of your own it behaves differently — see below.

The transaction covers what runs on :repo. A graph store holding its data anywhere else is outside it — that includes the built-in :memory backend, whose sweep is a GenServer.call, not only custom stores. So the two can disagree in both directions: a store that fails partway through its own sweep may have applied some of it even though the document survives, and a store that sweeps successfully before the repo side rolls back has dropped graph data for a document that is still there.

Inside a transaction of your own

It does not open one, and it does not roll anything back: rolling back a nested Ecto transaction aborts the outermost one, which would kill your transaction while handing you an error that looks recoverable.

So the guarantee above is weaker here. A :sweep_failed comes back as a tuple with your transaction intact, but the delete has been applied and stays in your transaction's scope — undoing it is yours to do. That is the one place where :sweep_failed does not mean "nothing happened".

A database failure is harsher: Postgres aborts to the nearest savepoint, and with none open that is the whole transaction, so the error ends it whatever gets returned. If you need to survive one, set your own savepoint around the call:

Repo.query!("SAVEPOINT before_delete")

case Arcana.delete(id, repo: Repo) do
  {:error, _reason} -> Repo.query!("ROLLBACK TO SAVEPOINT before_delete")
  :ok -> Repo.query!("RELEASE SAVEPOINT before_delete")
end

That recovers every failure the built-in stores can produce: a failed repo.delete, a :sweep_failed, and a sweep that raises.

A custom graph store can still defeat it, by opening a nested Ecto transaction of its own around work that then fails — DBConnection marks the connection aborted, and the ROLLBACK TO SAVEPOINT is refused along with everything else. Calling delete/2 outside your transaction avoids the question entirely.

Sweeping is optional for custom graph stores: one that doesn't implement Arcana.Graph.GraphStore.sweep_orphans/2 returns :ok and leaves the orphans alone.

Options

  • :repo - The Ecto repo to use (required)
  • :graph - Sweep orphaned graph data after deletion (default: from config)

embedder()

Returns the configured embedder as a {module, opts} tuple. See Arcana.Config for configuration options.

get_document(id, opts)

Fetches a document by id. See Arcana.Documents.get_document/2.

get_document_metadata(ids, opts)

Fetches sparse metadata for published documents in one explicitly scoped read. See Arcana.Documents.get_document_metadata/2.

graph_enabled?(opts)

Returns whether GraphRAG is enabled.

ingest(text, opts)

Ingests text content, creating a document with embedded chunks. See Arcana.Ingest.ingest/2 for options.

ingest_binary(binary, opts)

Ingests in-memory bytes, routing on the required :filename option's extension. See Arcana.Ingest.ingest_binary/2 for options.

ingest_file(path, opts)

Ingests a file, parsing its content and creating a document with embedded chunks. See Arcana.Ingest.ingest_file/2 for options.

list_documents(opts)

Lists documents, newest first. See Arcana.Documents.list_documents/1.

rewrite_query(query, opts \\ [])

Rewrites a query using a provided rewriter function. See Arcana.Search.rewrite_query/2 for options.

search(query, opts)

Searches for chunks similar to the query. See Arcana.Search.search/2 for options.