Ingests text content, creating a document with embedded chunks.

Options

• :repo - The Ecto repo to use (required)
• :source_id - An optional identifier for grouping/filtering
• :metadata - Optional map of metadata to store with the document
• :chunk_size - Maximum chunk size in characters (default: 1024)
• :chunk_overlap - Overlap between chunks (default: 200)
• :collection - Collection name (string) or map with name and description (default: "default")
• :graph - Enable GraphRAG extraction (default: from config)
• :replace - When true, atomically replaces any prior document with the same (collection, source_id) once the new ingest completes. Requires :source_id. See "Replacing documents" below.
• :on_chunk_error - What a chunk that fails to embed does to the rest of the document: :abort (default) or :skip. See "When a chunk fails to embed" below.

When a chunk fails to embed

Every chunk is embedded before anything is written, so a failure never leaves a partly-stored document behind.

With the default :abort, one rejected chunk fails the whole ingest and writes nothing:

{:error, {:embedding_failed, %{chunk_index: 12, reason: reason}}}

The chunk_index is there so you can find the offending input instead of guessing which of forty chunks the endpoint refused.

With :skip, the chunks that did embed are stored and the rest are reported:

{:ok, document, %{skipped_chunks: 1, reasons: [...], failed: [%{chunk_index: 12, reason: ...}]}}

For retrieval, 39 of 40 chunks beats losing the document. The reason is also written to document.error, so a partial ingest is still explicable from the row alone. Skipped chunks leave a gap in chunk_index rather than renumbering, since a chunk's "start_byte"/"end_byte" metadata is what locates it in the source.

If every chunk fails, :skip returns {:error, {:all_chunks_failed, failures}} and writes nothing: a document with no chunks is invisible to search but still counts in listings, which is the state this option exists to avoid.

Replacing documents

With replace: true, source_id becomes a stable document identity: re-ingesting the same identity supersedes earlier documents instead of accumulating next to them. The new document is ingested first and predecessors (including :failed/:processing leftovers from crashed attempts) are deleted only after it completes, so the old chunks stay searchable until the replacement lands and their rows cascade away with the document.

The swap runs in a transaction under a per-identity advisory lock, so callers don't need their own mutex for correctness. If two ingests for the same identity run concurrently, the first to complete wins and the other returns {:error, :replaced_by_concurrent_ingest} (or may fail while storing chunks whose document was already replaced).

A replacement that fails to embed leaves the predecessor untouched: the swap only runs once the new document's chunks are all in hand.

Ingests in-memory bytes, parsing them the same way ingest_file/2 parses a file on disk.

:filename is required: its extension picks the parser and its value is stored as the document's file_path for provenance. Nothing is written to disk.

Options

• :filename - Name the bytes came from, e.g. "report.docx" (required)
• :repo - The Ecto repo to use (required)
• :source_id - An optional identifier for grouping/filtering
• :metadata - Optional map of metadata to store with the document
• :chunk_size - Maximum chunk size in characters (default: 1024)
• :chunk_overlap - Overlap between chunks (default: 200)
• :collection - Collection name to organize the document (default: "default")
• :graph - Enable GraphRAG extraction (default: from config)
• :replace - When true, atomically replaces any prior document with the same (collection, source_id) once the new ingest completes. Requires :source_id. See "Replacing documents" in ingest/2.
• :on_chunk_error - :abort (default) or :skip. See "When a chunk fails to embed" in ingest/2.

Parsers that need a path

A parser only handles binaries when it says so via Arcana.FileParser.supports_binary?/0. The built-in PDF parser shells out to pdftotext and needs a real file, so ingesting PDF bytes returns {:error, {:binary_unsupported, Arcana.FileParser.PDF.Poppler}} — write them to a temp file and use ingest_file/2 instead.

A parser that is also unavailable reports that instead, since a retry through ingest_file/2 would fail just the same: with pdftotext missing, PDF bytes come back as {:error, :poppler_not_available}.

Ingests a file, parsing its content and creating a document with embedded chunks.

Handles plain text, markdown, and PDF natively, plus any format with a parser registered under config :arcana, :file_parsers — see Arcana.Parser for resolution and Arcana.FileParser for the behaviour. The document's content_type comes from the resolved parser.

Options

• :repo - The Ecto repo to use (required)
• :source_id - An optional identifier for grouping/filtering
• :metadata - Optional map of metadata to store with the document
• :chunk_size - Maximum chunk size in characters (default: 1024)
• :chunk_overlap - Overlap between chunks (default: 200)
• :collection - Collection name to organize the document (default: "default")
• :graph - Enable GraphRAG extraction (default: from config)
• :replace - When true, atomically replaces any prior document with the same (collection, source_id) once the new ingest completes. Requires :source_id. See "Replacing documents" in ingest/2.
• :on_chunk_error - :abort (default) or :skip. See "When a chunk fails to embed" in ingest/2.

Chunk metadata

Every chunk stores the "start_byte"/"end_byte" range it occupies in the extracted text. When the parser also reports page positions (the built-in PDF parser does), chunks additionally carry "page_start" and "page_end", so Arcana.search/2 results can cite a page:

[result | _] = Arcana.search("refund policy", repo: Repo)
result.metadata["page_start"]
#=> 4