RAG (Retrieval Augmented Generation) library for Elixir.
Arcana provides document ingestion, embedding, and vector search capabilities that you can embed into any Phoenix/Ecto application.
Usage
# Ingest a document
{:ok, document} = Arcana.ingest("Your text content", repo: MyApp.Repo)
# Search for relevant chunks
{:ok, results} = Arcana.search("your query", repo: MyApp.Repo)
# Ask questions with RAG
{:ok, answer} = Arcana.ask("What is X?", repo: MyApp.Repo, llm: my_llm)
# Delete a document
:ok = Arcana.delete(document.id, repo: MyApp.Repo)Modules
Arcana.Config- Configuration managementArcana.Ingest- Document ingestionArcana.Search- Vector and hybrid searchArcana.Ask- RAG question answeringArcana.Graph- GraphRAG functionality
Summary
Functions
Asks a question using retrieved context from the knowledge base.
See Arcana.Ask.ask/2 for options.
Returns the configured chunker as a {module, opts} tuple.
See Arcana.Config for configuration options.
Returns the current Arcana configuration.
Counts documents matching the list_documents/1 filters.
See Arcana.Documents.count_documents/1.
Deletes a document and all its chunks.
Returns the configured embedder as a {module, opts} tuple.
See Arcana.Config for configuration options.
Fetches a document by id. See Arcana.Documents.get_document/2.
Fetches sparse metadata for published documents in one explicitly scoped read.
See Arcana.Documents.get_document_metadata/2.
Returns whether GraphRAG is enabled.
Ingests text content, creating a document with embedded chunks.
See Arcana.Ingest.ingest/2 for options.
Ingests in-memory bytes, routing on the required :filename option's
extension. See Arcana.Ingest.ingest_binary/2 for options.
Ingests a file, parsing its content and creating a document with embedded chunks.
See Arcana.Ingest.ingest_file/2 for options.
Lists documents, newest first. See Arcana.Documents.list_documents/1.
Rewrites a query using a provided rewriter function.
See Arcana.Search.rewrite_query/2 for options.
Searches for chunks similar to the query.
See Arcana.Search.search/2 for options.
Functions
Asks a question using retrieved context from the knowledge base.
See Arcana.Ask.ask/2 for options.
Returns the configured chunker as a {module, opts} tuple.
See Arcana.Config for configuration options.
Returns the current Arcana configuration.
Counts documents matching the list_documents/1 filters.
See Arcana.Documents.count_documents/1.
Deletes a document and all its chunks.
When the graph is enabled, also sweeps the document's collection for orphaned graph data: entities left with zero mentions are deleted and communities that referenced them are marked dirty so the next summarize pass regenerates them.
Returns :ok, {:error, :not_found}, {:error, {:sweep_failed, reason}}
when the graph store returns an error from its sweep, or
{:error, reason} for a database failure such as a foreign key violation,
or a not-null violation from another table pointing at the document.
A sweep that raises a database error reports as a database failure rather
than as :sweep_failed — so match {:error, _} as well if you route on the
sweep case. sweep_orphans/2 deletes through repo.delete_all/1, which
skips changeset constraint mapping, so a host row referencing an entity the
sweep wants to remove arrives that way.
When a non-Ecto graph store, including the built-in :memory store, is
cleaned after the database delete commits, a returned error or exception is
wrapped as
{:error, {:post_commit_graph_cleanup_failed, context}}. The context carries
:reason, :chunk_ids, :published_chunk_ids, and :collection_id, so the
graph deletion and sweep can be retried even though the document row is
already gone. Calling delete/2 inside an existing Ecto transaction when an
a non-Ecto graph cleanup is required returns
{:error, :external_graph_store_requires_post_commit_delete} before deleting.
A concurrent delete reports {:error, :not_found}, the same as losing that
race by a moment more would have.
Infrastructure failures — a lost connection, a pool timeout — still raise. There is nothing useful to hand back and the transaction's fate is unknown, so those are left to the caller's supervisor rather than flattened into a tuple that would read like a clean refusal.
Called on its own, the delete and the orphan sweep run in one transaction,
so a :sweep_failed leaves the document in place: nothing happened and the
call can be retried. That is worth knowing if you are upgrading, because
this case used to delete the document and report the failed cleanup
afterwards. Inside a transaction of your own it behaves differently — see
below.
The transaction covers what runs on :repo. A graph store holding its data
anywhere else is outside it — that includes the built-in :memory backend,
whose sweep is a GenServer.call, not only custom stores. So the two can
disagree in both directions: a store that fails partway through its own
sweep may have applied some of it even though the document survives, and a
store that sweeps successfully before the repo side rolls back has dropped
graph data for a document that is still there.
Inside a transaction of your own
It does not open one, and it does not roll anything back: rolling back a nested Ecto transaction aborts the outermost one, which would kill your transaction while handing you an error that looks recoverable.
So the guarantee above is weaker here. A :sweep_failed comes back as a
tuple with your transaction intact, but the delete has been applied and
stays in your transaction's scope — undoing it is yours to do. That is the
one place where :sweep_failed does not mean "nothing happened".
A database failure is harsher: Postgres aborts to the nearest savepoint, and with none open that is the whole transaction, so the error ends it whatever gets returned. If you need to survive one, set your own savepoint around the call:
Repo.query!("SAVEPOINT before_delete")
case Arcana.delete(id, repo: Repo) do
{:error, _reason} -> Repo.query!("ROLLBACK TO SAVEPOINT before_delete")
:ok -> Repo.query!("RELEASE SAVEPOINT before_delete")
endThat recovers every failure the built-in stores can produce: a failed
repo.delete, a :sweep_failed, and a sweep that raises.
A custom graph store can still defeat it, by opening a nested Ecto
transaction of its own around work that then fails — DBConnection marks the
connection aborted, and the ROLLBACK TO SAVEPOINT is refused along with
everything else. Calling delete/2 outside your transaction avoids the
question entirely.
Sweeping is optional for custom graph stores: one that doesn't
implement Arcana.Graph.GraphStore.sweep_orphans/2 returns :ok and
leaves the orphans alone.
Options
:repo- The Ecto repo to use (required):graph- Sweep orphaned graph data after deletion (default: from config)
Returns the configured embedder as a {module, opts} tuple.
See Arcana.Config for configuration options.
Fetches a document by id. See Arcana.Documents.get_document/2.
Fetches sparse metadata for published documents in one explicitly scoped read.
See Arcana.Documents.get_document_metadata/2.
Returns whether GraphRAG is enabled.
Ingests text content, creating a document with embedded chunks.
See Arcana.Ingest.ingest/2 for options.
Ingests in-memory bytes, routing on the required :filename option's
extension. See Arcana.Ingest.ingest_binary/2 for options.
Ingests a file, parsing its content and creating a document with embedded chunks.
See Arcana.Ingest.ingest_file/2 for options.
Lists documents, newest first. See Arcana.Documents.list_documents/1.
Rewrites a query using a provided rewriter function.
See Arcana.Search.rewrite_query/2 for options.
Searches for chunks similar to the query.
See Arcana.Search.search/2 for options.