View Source Torus (Torus v0.2.0)
Torus is a plug-and-play library that seamlessly integrates PostgreSQL's search into Ecto, streamlining the construction of advanced search queries.
Usage
The package can be installed by adding torus
to your list of dependencies in mix.exs
:
def deps do
[
{:torus, "~> 0.2.0"}
]
end
Then, in any query, you can (for example) add a full-text search:
Post
|> Torus.full_text_dynamic([p], [p.title, p.body], "uncovered hogwarts")
|> Repo.all()
See full_text_dynamic/5
for more details.
5 types of searches:
Similarity: Searches for items that are closely alike based on attributes, often using measures like cosine similarity or Euclidean distance.
See
similarity/5
for more details.Text Search Vectors: Uses term-document matrix vectors for full-text search, enabling efficient querying and ranking based on term frequency. - PostgreSQL: Documentation: 17: Chapter 12. Full Text Search
SELECT to_tsvector('english', 'The quick brown fox jumps over the lazy dog') @@ to_tsquery('fox & dog');
See
full_text_dynamic/5
for more details.Semantic Search: Understands the contextual meaning of queries to match and retrieve related content, often utilizing natural language processing. Semantic Search with PostgreSQL and OpenAI Embeddings | by Dima Timofeev | Towards Data Science
Hybrid Search: Combines multiple search techniques (e.g., keyword and semantic) to leverage their strengths for more accurate results.
3rd Party Engines/Providers: Utilizes external services or software specifically designed for optimized and scalable search capabilities, such as Elasticsearch or Algolia.
Torus support
For now, Torus supports similarity and full-text search, with plans to expand support further. These docs will be updated with more examples on which search type to choose and how to make them more performant (by adding indexes or using specific functions).
Summary
Functions
Full text search with rank ordering. Accepts a list of columns to search in. Cleans the term, so it can be input directly by the user.
Wrapper around postgres ilike
function. Accepts a list of columns to search in.
Wrapper around postgres similarity
function. Accepts a list of columns to search in.
Functions
Full text search with rank ordering. Accepts a list of columns to search in. Cleans the term, so it can be input directly by the user.
Example usage
iex> insert_post!(title: "Hogwarts Shocker", body: "A spell disrupts the Quidditch Cup.")
...> insert_post!(title: "Diagon Bombshell", body: "Secrets uncovered in the heart of Hogwarts.")
...> insert_post!(title: "Completely unrelated", body: "No magic here!")
...> Post
...> |> Torus.full_text_dynamic([p], [p.title, p.body], "uncovered hogwarts")
...> |> select([p], p.title)
...> |> Repo.all()
["Diagon Bombshell"]
TODO: Add section on optimization, tradeoffs, etc.
Wrapper around postgres ilike
function. Accepts a list of columns to search in.
Doesn't clean the term, so it needs to be sanitized before being passed in.
Examples
iex> insert_post!(title: "Hogwarts Shocker", body: "A spell disrupts the Quidditch Cup.")
...> insert_post!(title: "Diagon Bombshell", body: "Secrets uncovered in the heart of Hogwarts.")
...> insert_post!(title: "Completely unrelated", body: "No magic here!")
...> Post
...> |> Torus.ilike([p], [p.title, p.body], "%ogw%")
...> |> select([p], p.title)
...> |> order_by(:id)
...> |> Repo.all()
["Hogwarts Shocker", "Diagon Bombshell"]
TODO: Add section on optimization, tradeoffs, etc.
Wrapper around postgres similarity
function. Accepts a list of columns to search in.
You need to have pg_trgm extension installed.
Examples
iex> insert_post!(title: "Hogwarts Shocker", body: "A spell disrupts the Quidditch Cup.")
...> insert_post!(title: "Diagon Bombshell", body: "Secrets uncovered in the heart of Hogwarts.")
...> insert_post!(title: "Completely unrelated", body: "No magic here!")
...> Post
...> |> Torus.similarity([p], [p.title, p.body], "boshel", limit: 1)
...> |> select([p], p.title)
...> |> Repo.all()
["Diagon Bombshell"]
Similarity search in postgres.
TODO: Add section on optimization, tradeoffs, etc.