LangEx.Embedding.Hashing (LangEx v0.11.3)

Copy Markdown View Source

Dependency-free text embedder using the hashing trick.

Tokenizes text and hashes each token into one of :dims buckets, producing a fixed-length term-frequency vector. Cosine similarity over these vectors approximates lexical overlap, so it needs no model and no network — a sensible default embedder for LangEx.Store semantic search when a neural embedding provider is not configured:

Graph.compile(builder,
  store: {LangEx.Store.ETS, index: [embed: &LangEx.Embedding.Hashing.embed/1]}
)

It captures word overlap, not meaning: "db is slow" and "database latency" share no tokens and score near zero. Supply a neural embedder when semantic (meaning-based) similarity matters.

Summary

Functions

Embeds text into a fixed-length term-frequency vector.

Functions

embed(text, opts \\ [])

@spec embed(
  String.t(),
  keyword()
) :: [float()]

Embeds text into a fixed-length term-frequency vector.

Options:

  • :dims - vector length / number of hash buckets (default 256)