TiDB.Vector (tidb v0.1.0)

Copy Markdown View Source

A vector struct and utility module for TiDB vector operations.

TiDB stores vector embeddings as arrays of 32-bit floating-point numbers. TiDB.Vector encapsulates the binary representation and dimension count of these vectors, providing high-performance encoding and decoding.

Struct Fields

  • :data - Compact binary storage of 32-bit floating-point values in big-endian format.
  • :dim - Non-negative integer representing the number of vector dimensions (elements).

Limits

TiDB supports vectors up to a maximum dimension of 16,383. Attempting to create a vector exceeding this limit will raise an ArgumentError.

Nx Interoperability

When the optional :nx dependency is included in your project, TiDB.Vector allows seamless conversion from and to 1D Nx.Tensor structs:

# Creating from a 1D Tensor
tensor = Nx.tensor([1.0, 2.0, 3.0], type: :f32)
vector = TiDB.Vector.new(tensor)

# Converting back to an Nx Tensor
Nx.Tensor.t() = TiDB.Vector.to_tensor(vector)

Summary

Types

t()

A %TiDB.Vector{} struct holding binary 32-bit float vector data and dimension size.

Functions

from_binary(binary)

Creates a %TiDB.Vector{} from a raw binary string of 32-bit floating point numbers.

from_string(string)

Parses a %TiDB.Vector{} from a JSON-formatted array string representation returned by TiDB.

new(list)

Creates a %TiDB.Vector{} from a list of numbers, a string literal, an Nx.Tensor, or returns the vector as-is.

to_binary(vector)

Extracts the underlying raw binary representation from a %TiDB.Vector{} struct.

to_list(vector)

Converts a %TiDB.Vector{} struct to a list of floating-point numbers rounded to 6 decimal places.

to_string(vector)

Formats a %TiDB.Vector{} into a compact TiDB SQL string literal representation (e.g. "[1.0,2.0]").

to_tensor(vector)

Converts a %TiDB.Vector{} to an Nx.Tensor with :f32 type.

Types

t()

@type t() :: %TiDB.Vector{data: binary(), dim: non_neg_integer()}

A %TiDB.Vector{} struct holding binary 32-bit float vector data and dimension size.

Functions

from_binary(binary)

@spec from_binary(binary()) :: t()

Creates a %TiDB.Vector{} from a raw binary string of 32-bit floating point numbers.

Arguments

  • binary - A binary whose byte size is a multiple of 4 (each 32-bit float occupies 4 bytes).

Examples

iex> bin = <<1.0::float-32, 2.0::float-32>>
iex> TiDB.Vector.from_binary(bin)
#TiDB.Vector<[1.0, 2.0]>

Errors

Raises ArgumentError if binary byte size is not a multiple of 4.

from_string(string)

@spec from_string(String.t()) :: t()

Parses a %TiDB.Vector{} from a JSON-formatted array string representation returned by TiDB.

Arguments

  • string - A string such as "[1.0, 2.0, 3.0]" or "[1, 2, 3]".

Examples

iex> TiDB.Vector.from_string("[0.1, 0.2, 0.3]")
#TiDB.Vector<[0.1, 0.2, 0.3]>

Errors

Raises ArgumentError if string is not valid JSON or contains non-numeric values.

new(list)

@spec new([number()] | String.t() | t() | struct()) :: t()

Creates a %TiDB.Vector{} from a list of numbers, a string literal, an Nx.Tensor, or returns the vector as-is.

Arguments

  • input - Can be one of:
    • list(number()) - A list of floats or integers (e.g. [1.0, 2.0, 3.0] or [1, 2, 3]).
    • String.t() - A JSON-formatted array string (e.g. "[1.0, 2.0, 3.0]").
    • %TiDB.Vector{} - An existing vector struct (returned unchanged).
    • Nx.Tensor.t() - A 1D (rank-1) tensor (requires optional :nx dependency).

Examples

iex> TiDB.Vector.new([1.0, 2.0, 3.0])
#TiDB.Vector<[1.0, 2.0, 3.0]>

iex> TiDB.Vector.new("[0.5, -1.2, 3.4]")
#TiDB.Vector<[0.5, -1.2, 3.4]>

iex> vec = TiDB.Vector.new([1.0, 2.0])
iex> TiDB.Vector.new(vec) == vec
true

Errors

Raises ArgumentError if:

  • Dimension count exceeds maximum limit (16,383).
  • List contains non-numeric elements.
  • String format cannot be parsed as a valid numeric array.
  • An Nx.Tensor does not have a rank of 1.
  • Argument is of an unsupported type.

to_binary(vector)

@spec to_binary(t()) :: binary()

Extracts the underlying raw binary representation from a %TiDB.Vector{} struct.

Arguments

  • vector - A %TiDB.Vector{} struct.

Examples

iex> vec = TiDB.Vector.new([1.0, 2.0])
iex> TiDB.Vector.to_binary(vec)
<<63, 128, 0, 0, 64, 0, 0, 0>>

to_list(vector)

@spec to_list(t()) :: [float()]

Converts a %TiDB.Vector{} struct to a list of floating-point numbers rounded to 6 decimal places.

Arguments

  • vector - A %TiDB.Vector{} struct.

Examples

iex> vec = TiDB.Vector.new([1.0, 2.5, 3.75])
iex> TiDB.Vector.to_list(vec)
[1.0, 2.5, 3.75]

to_string(vector)

@spec to_string(t()) :: String.t()

Formats a %TiDB.Vector{} into a compact TiDB SQL string literal representation (e.g. "[1.0,2.0]").

Arguments

  • vector - A %TiDB.Vector{} struct.

Examples

iex> vec = TiDB.Vector.new([1.0, 2.0, 3.0])
iex> TiDB.Vector.to_string(vec)
"[1.0,2.0,3.0]"

to_tensor(vector)

@spec to_tensor(t()) :: Nx.Tensor.t()

Converts a %TiDB.Vector{} to an Nx.Tensor with :f32 type.

Note: This function is only defined when :nx is available.

Arguments

  • vector - A %TiDB.Vector{} struct.

Examples

iex> vec = TiDB.Vector.new([1.0, 2.0, 3.0])
iex> TiDB.Vector.to_tensor(vec)
#Nx.Tensor<
  f32[3]
  [1.0, 2.0, 3.0]
>