TiDB (tidb v0.1.0)

TiDB Vector Search and Ecto integration for Elixir applications.

TiDB provides tools and extensions for utilizing TiDB's native vector search and advanced database features within Elixir and Ecto ecosystems.

Features

• Vector Struct & Serialization (TiDB.Vector): A compact 32-bit floating point vector representation with conversions to/from Elixir lists, binaries, TiDB string literals, and optional Nx tensors.
• Ecto Custom Type (TiDB.Ecto.Vector): Seamlessly map TiDB VECTOR columns to %TiDB.Vector{} structs in your Ecto schemas.
• Vector Query Macros (TiDB.Ecto.Vector.Query): Ecto query helper macros for TiDB vector distance functions including Cosine distance (vec_cosine_distance/2), L2/Euclidean distance (vec_l2_distance/2), L1/Manhattan distance (vec_l1_distance/2), negative inner product (vec_negative_inner_product/2), vector dimensions (vec_dims/1), and L2 norm (vec_l2_norm/1).
• Migration Helpers (TiDB.Ecto.Migrations): Migration macros to configure TiFlash replicas (enable_tiflash/2, disable_tiflash/1) and create Vector (vector_index/3) or Full-Text (fulltext_index/3) indexes with columnar replicas on demand.
• Full-Text Search Macros (TiDB.Ecto.Fulltext.Query): Query helpers for TiDB full-text search.

Quickstart

1. Define a Migration

defmodule MyApp.Repo.Migrations.CreateDocuments do
  use Ecto.Migration
  import TiDB.Ecto.Migrations

  def change do
    create table(:documents) do
      add :title, :string
      add :content, :text
      add :embedding, :vector, size: 384

      timestamps()
    end

    # Enable TiFlash replica and create an HNSW vector index
    enable_tiflash("documents")
    vector_index("documents", "embedding", distance: :cosine)
  end
end

2. Define an Ecto Schema

defmodule MyApp.Document do
  use Ecto.Schema
  import Ecto.Changeset

  schema "documents" do
    field :title, :string
    field :content, :string
    field :embedding, TiDB.Ecto.Vector

    timestamps()
  end

  def changeset(document, attrs) do
    document
    |> cast(attrs, [:title, :content, :embedding])
    |> validate_required([:title, :content, :embedding])
  end
end

3. Insert Vector Data

Vectors can be supplied as a list of numbers, an %TiDB.Vector{} struct, or an Nx.Tensor:

# Using a list of numbers
%MyApp.Document{}
|> MyApp.Document.changeset(%{
  title: "TiDB Vector Guide",
  content: "Vector search in TiDB...",
  embedding: [0.023, -0.125, 0.891]
})
|> MyApp.Repo.insert!()

# Using TiDB.Vector explicitly
vector = TiDB.Vector.new([0.023, -0.125, 0.891])
MyApp.Repo.insert!(%MyApp.Document{title: "Doc 2", embedding: vector})

4. Query by Vector Similarity (k-NN / ANN)

import Ecto.Query
import TiDB.Ecto.Vector.Query

query_vector = TiDB.Vector.new([0.025, -0.120, 0.880])

# Find the 5 most similar documents using Cosine Distance
results =
  from(d in MyApp.Document,
    select: %{
      id: d.id,
      title: d.title,
      distance: vec_cosine_distance(d.embedding, ^query_vector)
    },
    order_by: [asc: vec_cosine_distance(d.embedding, ^query_vector)],
    limit: 5
  )
  |> MyApp.Repo.all()