View Source Torus (Torus v0.2.0)

Torus is a plug-and-play library that seamlessly integrates PostgreSQL's search into Ecto, streamlining the construction of advanced search queries.

Usage

The package can be installed by adding torus to your list of dependencies in mix.exs:

def deps do
  [
    {:torus, "~> 0.2.0"}
  ]
end

Then, in any query, you can (for example) add a full-text search:

Post
|> Torus.full_text_dynamic([p], [p.title, p.body], "uncovered hogwarts")
|> Repo.all()

See full_text_dynamic/5 for more details.

5 types of searches:

  1. Similarity: Searches for items that are closely alike based on attributes, often using measures like cosine similarity or Euclidean distance.

    See similarity/5 for more details.

  2. Text Search Vectors: Uses term-document matrix vectors for full-text search, enabling efficient querying and ranking based on term frequency. - PostgreSQL: Documentation: 17: Chapter 12. Full Text Search

    SELECT to_tsvector('english', 'The quick brown fox jumps over the lazy dog') @@ to_tsquery('fox & dog');

    See full_text_dynamic/5 for more details.

  3. Semantic Search: Understands the contextual meaning of queries to match and retrieve related content, often utilizing natural language processing. Semantic Search with PostgreSQL and OpenAI Embeddings | by Dima Timofeev | Towards Data Science

  4. Hybrid Search: Combines multiple search techniques (e.g., keyword and semantic) to leverage their strengths for more accurate results.

  5. 3rd Party Engines/Providers: Utilizes external services or software specifically designed for optimized and scalable search capabilities, such as Elasticsearch or Algolia.

Torus support

For now, Torus supports similarity and full-text search, with plans to expand support further. These docs will be updated with more examples on which search type to choose and how to make them more performant (by adding indexes or using specific functions).

Summary

Functions

Full text search with rank ordering. Accepts a list of columns to search in. Cleans the term, so it can be input directly by the user.

Wrapper around postgres ilike function. Accepts a list of columns to search in.

Wrapper around postgres similarity function. Accepts a list of columns to search in.

Functions

full_text_dynamic(query, bindings, qualifiers, term, args \\ [])

(macro)

Full text search with rank ordering. Accepts a list of columns to search in. Cleans the term, so it can be input directly by the user.

Example usage

iex> insert_post!(title: "Hogwarts Shocker", body: "A spell disrupts the Quidditch Cup.")
...> insert_post!(title: "Diagon Bombshell", body: "Secrets uncovered in the heart of Hogwarts.")
...> insert_post!(title: "Completely unrelated", body: "No magic here!")
...>  Post
...> |> Torus.full_text_dynamic([p], [p.title, p.body], "uncovered hogwarts")
...> |> select([p], p.title)
...> |> Repo.all()
["Diagon Bombshell"]

TODO: Add section on optimization, tradeoffs, etc.

ilike(query, bindings, qualifiers, term, args \\ [])

(macro)

Wrapper around postgres ilike function. Accepts a list of columns to search in.

Doesn't clean the term, so it needs to be sanitized before being passed in.

Examples

iex> insert_post!(title: "Hogwarts Shocker", body: "A spell disrupts the Quidditch Cup.")
...> insert_post!(title: "Diagon Bombshell", body: "Secrets uncovered in the heart of Hogwarts.")
...> insert_post!(title: "Completely unrelated", body: "No magic here!")
...> Post
...> |> Torus.ilike([p], [p.title, p.body], "%ogw%")
...> |> select([p], p.title)
...> |> order_by(:id)
...> |> Repo.all()
["Hogwarts Shocker", "Diagon Bombshell"]

TODO: Add section on optimization, tradeoffs, etc.

similarity(query, bindings, qualifiers, term, args \\ [])

(macro)

Wrapper around postgres similarity function. Accepts a list of columns to search in.

You need to have pg_trgm extension installed.

Examples

iex> insert_post!(title: "Hogwarts Shocker", body: "A spell disrupts the Quidditch Cup.")
...> insert_post!(title: "Diagon Bombshell", body: "Secrets uncovered in the heart of Hogwarts.")
...> insert_post!(title: "Completely unrelated", body: "No magic here!")
...>  Post
...> |> Torus.similarity([p], [p.title, p.body], "boshel", limit: 1)
...> |> select([p], p.title)
...> |> Repo.all()
["Diagon Bombshell"]

Similarity search in postgres.

TODO: Add section on optimization, tradeoffs, etc.