Arcana.VectorStore.Memory (Arcana v2.0.1)

Copy Markdown View Source

In-memory vector store using HNSWLib for approximate nearest neighbor search.

Useful for:

  • Testing embedding models without database migrations
  • Smaller RAGs where pgvector overhead isn't justified
  • Development and experimentation workflows

Usage

# Start the server
{:ok, pid} = Arcana.VectorStore.Memory.start_link(name: MyApp.VectorStore)

# Store vectors
:ok = Memory.store(pid, "default", "chunk-1", embedding, %{text: "hello"})

# Search
results = Memory.search(pid, "default", query_embedding, limit: 10)

# Delete
:ok = Memory.delete(pid, "default", "chunk-1")

# Clear collection
:ok = Memory.clear(pid, "default")

Requirements

Requires the hnswlib dependency.

Notes

  • Data is not persisted to disk - all vectors are lost when the process stops
  • Uses cosine similarity for semantic search
  • Recommended for < 100K vectors per collection

Summary

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

clear(server, collection)

Clears all vectors from a collection.

delete(server, collection, id)

Deletes a vector from a collection.

search(server, collection, query_embedding, opts \\ [])

Searches for similar vectors in a collection.

search_text(server, collection, query_text, opts \\ [])

Searches for matching text in a collection (fulltext search).

start_link(opts \\ [])

Starts the Memory vector store GenServer.

store(server, collection, id, embedding, metadata)

Stores a vector with its id and metadata in a collection.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

clear(server, collection)

Clears all vectors from a collection.

Parameters

  • server - The GenServer pid or name
  • collection - The collection name to clear

Returns

  • :ok on success

delete(server, collection, id)

Deletes a vector from a collection.

Parameters

  • server - The GenServer pid or name
  • collection - The collection name
  • id - The vector's unique identifier

Returns

  • :ok on success
  • {:error, :not_found} if the id doesn't exist in the collection

search(server, collection, query_embedding, opts \\ [])

Searches for similar vectors in a collection.

Parameters

  • server - The GenServer pid or name
  • collection - The collection name to search in
  • query_embedding - The query vector as a list of floats
  • opts - Search options
    • :limit - Maximum number of results to return (default: 10)

Returns

A list of maps with keys:

  • :id - The vector's unique identifier
  • :metadata - The associated metadata map
  • :score - Similarity score (0.0 to 1.0, higher is more similar)

search_text(server, collection, query_text, opts \\ [])

Searches for matching text in a collection (fulltext search).

Uses simple term matching with TF-IDF-like scoring.

Parameters

  • server - The GenServer pid or name
  • collection - The collection name to search in
  • query_text - The query string
  • opts - Search options
    • :limit - Maximum number of results to return (default: 10)

Returns

A list of maps with keys:

  • :id - The vector's unique identifier
  • :metadata - The associated metadata map
  • :score - Relevance score based on term matching (higher is more relevant)

start_link(opts \\ [])

Starts the Memory vector store GenServer.

Options

  • :name - The name to register the GenServer under (optional)
  • :max_elements - Maximum number of elements per collection (default: 10,000)

store(server, collection, id, embedding, metadata)

Stores a vector with its id and metadata in a collection.

Parameters

  • server - The GenServer pid or name
  • collection - The collection name (e.g., "default", "products")
  • id - Unique identifier for the vector
  • embedding - The embedding vector as a list of floats
  • metadata - A map of metadata associated with the vector

Returns

  • :ok on success