Refine (Refine v0.3.0)

Refine is an Elixir library for fast faceted search.

Summary

Facets

Creates a facets table based on a source table and populates it with bitmap data, facet labels, and facet values.

create_facets_table_if_not_exists(config, opts \\ [])

Identical to create_facets_table/2 but in case the table already exists, skips table creation silently without returning an error.

drop_facets_table(config, opts \\ [])

Removes the facets table and all installed PostgreSQL triggers and functions.

merge_deltas(config, options \\ [])

Aggregates and applies membership changes from the deltas table to update the facets table.

Search

search(config, options \\ [])

Performs a search on the source table using an optional base query and selected facet values.

Helpers

table_exists?(qualified_table_name, repo)

Tests whether a table exists. Pass a qualified table name (including schema prefix) if the table is not in the "public" schema.

Types

create_facets_table_return()

database_option()

database_options()

facet_config()

facet_configs()

facets_table_config()

join_config()

join_configs()

order_by_entry()

order_direction()

postgrex_option()

postgrex_options()

repo()

search_option()

search_options()

search_return()

search_return_data()

sort_nulls()

Facets

create_facets_table(config, opts \\ [])

@spec create_facets_table(facets_table_config(), database_options()) ::
  create_facets_table_return()

Creates a facets table based on a source table and populates it with bitmap data, facet labels, and facet values.

Alongside the creation of the table:

Creates the roaringbitmap extension (if it isn't created already).
With config option add_identity_column_if_not_exists: adds an identity column to the source table (if no valid integer ID column is found in the table).

Configuration

See: Facets table configuration

Options

repo - The Ecto Repo module that contains a Postgres adapter. Omit when the repo is configured globally.
timeout - Postgrex timeout option.

Examples

config = %{
  facets_table: "articles_facets",
  source_table: "articles",
  add_identity_column_if_not_exists: true,
  identity_column: "identity",
  facets: [
    %{facet_name: "draft"}
  ]
}

Refine.create_facets_table(config,
  repo: MyApp.Repo
)
{:ok, "articles_facets"}

To create a the facets table in a non-default schema:.

config = %{
  facets_table: "classifications.categories_facets",
  source_table: "classifications.categories",
  ...
}

Returns an error if the table exists:

Refine.create_facets_table(config,
  repo: MyApp.Repo
)
{:error, :facets_table_exists, "Table 'articles_facets' exists. ..."}

To recreate the facets table:

Refine.drop_facets_table(config,
  repo: MyApp.Repo
)

Refine.create_facets_table(config,
  repo: MyApp.Repo
)

create_facets_table_if_not_exists(config, opts \\ [])

@spec create_facets_table_if_not_exists(
  facets_table_config(),
  [database_option()]
) :: create_facets_table_return()

Identical to create_facets_table/2 but in case the table already exists, skips table creation silently without returning an error.

If you need to recreate an existing facets table, first call drop_facets_table/2. See also: create_facets_table/2.

Options:

repo - The Ecto Repo module that contains a Postgres adapter. Omit when the repo is configured globally.
timeout - Postgrex timeout option.

drop_facets_table(config, opts \\ [])

@spec drop_facets_table(facets_table_config(), database_options()) ::
  {:ok, String.t()} | {:error, :table_not_found} | {:error, Exception.t()}

Removes the facets table and all installed PostgreSQL triggers and functions.

Options:

repo - The Ecto Repo module that contains a Postgres adapter. Omit when the repo is configured globally.
timeout - Postgrex timeout option.

Examples

Refine.drop_facets_table(config, repo: MyApp.Repo)
{:ok, "articles_facets"}

merge_deltas(config, options \\ [])

@spec merge_deltas(facets_table_config(), database_options()) ::
  :ok | {:error, [Exception.t()]}

Aggregates and applies membership changes from the deltas table to update the facets table.

Search

search(config, options \\ [])

@spec search(facets_table_config(), search_options()) :: search_return()

Performs a search on the source table using an optional base query and selected facet values.

Parameter config is the configuration map used to create the facets table. Here it is used to read references to the source and facet tables.

Parameter options may include a base query, selected facet values, and fields to return in the results.

Examples

Return unfiltered rows in the source database:

Refine.search(config)

Apply pagination with limit and offset:

Refine.search(config, limit: 10, offset: 10)

Filter by facet values:

Refine.search(config,
  facets: %{"draft" => ["true"]},
  result_fields: ["id", "title"]
)

Filter a query by facet values:

query = from a in Article,
  where: ilike(a.title, ^"%memory%"),
  select: %{id: a.id, title: a.title}
facets = %{"draft" => ["true"]}

Refine.search(config,
  query: query,
  facets: facets
)

Options

query

An Ecto query applied to the source table to filter or shape the data.

Examples

With where filter and select fields:

query = from a in Article,
  where: ilike(a.title, ^"%memory%"),
  select: %{id: a.id, title: a.title, draft: a.draft}

For further options, see: Ecto.Query ➚.

A map containing the selected values for each facet.

The map keys are facet names (as defined in the configuration), while the values are the selected option values stored in the facets table.

Facet values are always strings.

Examples

%{
  "draft" => ["true"],
  "color" => ["blue", "red"]
}

If the list of facet options contains more than 1 value (blue and red), the filter on that facet is performed with an OR operator. In the example above, the logic becomes: draft=true AND (color=blue OR color=red).

order_by

Sorts the search results.

order_by takes a list of tuples, each in one of two forms:

{column, direction}
{column, direction, null_sort}

Where:

Value column is the column name (a string).
Value direction is :asc or :desc.
Value null_sort defines how null values are placed: :first, :last or :default. With :default (or the two-element form), the database default applies - nulls last for :asc, nulls first for :desc.

You can sort by:

Any column of the source table, whether or not it is included in the query's select
Any field included in the query's select, including joined fields (referenced by their select key)

Sorting by a column that is neither a source-table column nor a selected field raises an error.

Results are always ordered by the identity column as a final tiebreaker, so the ordering is stable even when the sorted columns contain duplicate values.

Examples

Sort by a single source column, descending:

Refine.search(config,
  order_by: [{"updated_at", :desc}],
  repo: Repo
)

Sort by multiple source columns:

order_by: [{"updated_at", :desc}, {"title", :asc}]

Sort by a source column that isn't selected:

query = from a in Article, select: %{id: a.id, title: a.title}

Refine.search(config,
  query: query,
  order_by: [{"updated_at", :desc}]
  repo: Repo,
)

Sort by a joined and selected field, referenced by its select key:

query =
  from a in Article,
    join: c in assoc(a, :category),
    select: %{id: a.id, category_name: c.name}

Refine.search(config,
  query: query,
  order_by: [{"category_name", :asc}],
  repo: Repo
)

Sort with null values first:

order_by: [{"title", :asc, :first}]
order_by: [{"title", :desc, :first}]

Sort with null values last:

order_by: [{"title", :asc, :last}]
order_by: [{"title", :desc, :last}]

Default null placement depends on direction:

order_by: [{"title", :asc}]   # nulls last (Postgres default for :asc)
order_by: [{"title", :desc}]  # nulls first (Postgres default for :desc)

limit

Limits the number of rows returned. Default: 10.

Must be combined with offset.

offset

The number of skipped rows. Default: 0.

Must be combined with limit.

result_fields

List of source fields to include in the result list. Use this to limit returned data when not using query, or to include columns not defined in the source Ecto schema.

result_fields takes precedence over any select expression in query.

Examples

result_fields = ["identity", "title"]

repo

The Ecto Repo module that contains a Postgres adapter. This can be omitted when the repo is configured globally - see: Installation.

timeout

Postgrex timeout option.

Helpers

table_exists?(qualified_table_name, repo)

@spec table_exists?(String.t(), repo()) :: boolean()

Tests whether a table exists. Pass a qualified table name (including schema prefix) if the table is not in the "public" schema.

Performs a database query using pg_class.

Examples

Refine.table_exists?("articles_facets", MyApp.Repo)

Refine.table_exists?("classifications.tags", MyApp.Repo)

Types

create_facets_table_return()

@type create_facets_table_return() ::
  {:ok, String.t()}
  | {:error, :column_not_found, String.t()}
  | {:error, :column_names_not_unique}
  | {:error, :facets_table_exists, String.t()}
  | {:error, :identity_column_already_exists, String.t()}
  | {:error, :identity_column_invalid_type, String.t()}
  | {:error, :identity_column_not_found, String.t()}
  | {:error, :invalid_table_name, String.t()}
  | {:error, :table_names_not_unique}
  | {:error, Exception.t()}

database_option()

@type database_option() :: {:repo, repo()} | postgrex_option()

database_options()

@type database_options() :: [database_option()]

facets_table_config()

@type facets_table_config() ::
  %{
    add_identity_column_if_not_exists: boolean() | nil,
    facets_table: String.t(),
    facets: facet_configs(),
    identity_column: String.t(),
    joins: join_configs() | nil,
    roaringbitmap_type: String.t() | nil,
    source_table: String.t()
  }
  | keyword()

join_config()

@type join_config() :: %{
  table: String.t(),
  match: String.t(),
  to: String.t(),
  on: String.t() | nil
}

join_configs()

@type join_configs() :: [join_config()]

order_by_entry()

@type order_by_entry() ::
  {String.t(), order_direction()}
  | {String.t(), order_direction(), sort_nulls()}

order_direction()

@type order_direction() :: :asc | :desc

postgrex_option()

@type postgrex_option() :: {:timeout, integer() | :infinity} | {:log, boolean()}

postgrex_options()

@type postgrex_options() :: [postgrex_option()]

repo()

@type repo() :: module()

search_option()

@type search_option() ::
  {:facets, map()}
  | {:limit, integer()}
  | {:offset, integer()}
  | {:order_by, [order_by_entry()]}
  | {:query, Ecto.Query.t()}
  | {:repo, repo()}
  | {:result_fields, [String.t()]}
  | postgrex_option()

search_options()

@type search_options() :: [search_option()]

search_return()

@type search_return() :: {:ok, search_return_data()} | {:error, Exception.t()}

search_return_data()

@type search_return_data() :: %{
  facets: map(),
  rows: [map()],
  total_count: integer(),
  types: map(),
  flop_meta: map() | nil
}

sort_nulls()

@type sort_nulls() :: :first | :last | :default