Arcana.VectorStore behaviour (Arcana v2.0.1)

Behaviour and dispatch module for vector storage backends.

Arcana supports two vector storage backends:

:pgvector (default) - PostgreSQL with pgvector extension
:memory - In-memory storage using HNSWLib

Configuration

# config/config.exs

# Use pgvector (default)
config :arcana, vector_store: :pgvector

# Use in-memory storage
config :arcana, vector_store: :memory

In-Memory Backend

When using :memory, you need to start the Memory server in your supervision tree:

children = [
  MyApp.Repo,
  {Arcana.VectorStore.Memory, name: Arcana.VectorStore.Memory}
]

The Memory backend is useful for:

Testing embedding models without database migrations
Smaller RAGs where pgvector overhead isn't justified
Development and experimentation workflows

Note: Memory backend data is not persisted - all vectors are lost when the process stops.

Custom Backend

To implement a custom backend, create a module that implements the Arcana.VectorStore behaviour:

defmodule MyApp.CustomVectorStore do
  @behaviour Arcana.VectorStore

  @impl true
  def store(collection, id, embedding, metadata, opts) do
    # Your implementation
  end

  @impl true
  def search(collection, query_embedding, opts) do
    # Your implementation
  end

  @impl true
  def delete(collection, id, opts) do
    # Your implementation
  end

  @impl true
  def clear(collection, opts) do
    # Your implementation
  end
end

Then configure:

config :arcana, vector_store: MyApp.CustomVectorStore

Summary

Callbacks

clear(binary, opts)

Clears all vectors from a collection.

delete(binary, binary, opts)

Deletes a vector from a collection.

search(binary, list, opts)

Searches for similar vectors in a collection (semantic search).

search_text(binary, query, opts)

Searches for matching text in a collection (fulltext search).

store(binary, binary, list, map, opts)

Stores a vector with its id and metadata in a collection.

Functions

backend()

Returns the configured vector store backend.

clear(collection, opts \\ [])

Clears a collection using the configured backend.

delete(collection, id, opts \\ [])

Deletes a vector using the configured backend.

search(collection, query_embedding, opts \\ [])

Searches for similar vectors using the configured backend.

search_text(collection, query_text, opts \\ [])

Searches for matching text using the configured backend (fulltext search).

store(collection, id, embedding, metadata, opts \\ [])

Stores a vector using the configured backend.

Callbacks

clear(binary, opts)

@callback clear(binary(), opts :: keyword()) :: :ok

Clears all vectors from a collection.

delete(binary, binary, opts)

@callback delete(binary(), binary(), opts :: keyword()) :: :ok | {:error, term()}

Deletes a vector from a collection.

search(binary, list, opts)

@callback search(binary(), list(), opts :: keyword()) :: [map()]

Searches for similar vectors in a collection (semantic search).

Returns a list of results with :id, :metadata, and :score keys.

search_text(binary, query, opts)

@callback search_text(binary(), query :: String.t(), opts :: keyword()) :: [map()]

Searches for matching text in a collection (fulltext search).

Returns a list of results with :id, :metadata, and :score keys. Score represents relevance based on term matching.

store(binary, binary, list, map, opts)

@callback store(binary(), binary(), list(), map(), opts :: keyword()) ::
  :ok | {:error, term()}

Stores a vector with its id and metadata in a collection.

Functions

backend()

Returns the configured vector store backend.

Examples

iex> Arcana.VectorStore.backend()
:pgvector

clear(collection, opts \\ [])

Clears a collection using the configured backend.

Options

:vector_store - Override the configured backend (see store/5 for format)

delete(collection, id, opts \\ [])

Deletes a vector using the configured backend.

Options

:vector_store - Override the configured backend (see store/5 for format)

search(collection, query_embedding, opts \\ [])

Searches for similar vectors using the configured backend.

Options

:vector_store - Override the configured backend (see store/5 for format)
:limit - Maximum number of results (default: 10)

Examples

# Use global config
VectorStore.search("products", query_embedding, limit: 10)

# Override with memory backend
VectorStore.search("products", query_embedding,
  vector_store: {:memory, pid: memory_pid},
  limit: 10)

search_text(collection, query_text, opts \\ [])

Searches for matching text using the configured backend (fulltext search).

Options

:vector_store - Override the configured backend (see store/5 for format)
:limit - Maximum number of results (default: 10)

Examples

# Use global config
VectorStore.search_text("products", "organic coffee", limit: 10)

# Override with memory backend
VectorStore.search_text("products", "organic coffee",
  vector_store: {:memory, pid: memory_pid},
  limit: 10)

store(collection, id, embedding, metadata, opts \\ [])

Stores a vector using the configured backend.

Options

:vector_store - Override the configured backend. Can be:
- {:memory, pid: pid} - Use memory backend with specific server
- {:pgvector, repo: MyRepo} - Use pgvector with specific repo
- MyCustomModule - Use a custom module implementing the behaviour
:limit - Maximum number of results (default: 10)

Examples

# Use global config
VectorStore.store("products", "id", embedding, metadata)

# Override with memory backend
VectorStore.store("products", "id", embedding, metadata,
  vector_store: {:memory, pid: memory_pid})

# Override with pgvector backend
VectorStore.store("products", "id", embedding, metadata,
  vector_store: {:pgvector, repo: MyApp.Repo})