Merges ranked hit lists from several sources (SearXNG instances, search engines) into one list: de-duplicates by normalized URL, scores by reciprocal rank, boosts hits several engines agree on, and demotes hits that share no terms with the query.
For each merged hit:
- rank - the sum over the lists it appears in of
1 / (k + position), position counted from 1.k: 60is Reciprocal Rank Fusion (Cormack, Clarke and Büttcher, 2009);k: 0is the plain1 / positionsum metasearch2 uses, which favors the top few spots more strongly. - agreement -
1 + engine_boost * (distinct engines - 1), so a page that Bing, Brave and DuckDuckGo all return outranks one only Bing returned at the same position. - lexical -
(1 - lexical_weight) + lexical_weight * coverage, where coverage is the share of the query's terms found in the hit's title, URL and description. An engine that answers a blocked client with unrelated pages (Bing does) gets those pages pushed down rather than trusted.
The score is rank * agreement * lexical. Duplicates keep the title of
their best-placed copy, the longest description, and every engine.
Summary
Functions
The share of query's terms (lowercased words, stopwords dropped) that
appear in the hit's title, URL or description: 1.0 when the query has no
terms.
Fuses lists (each a list of hits, best first) into one ranked list.
The mean coverage/2 of hits' first depth (default 5): how much of
the query a result list talks about. 0.0 for no hits.
The de-duplication key for url: scheme-less, lowercase host without
www., no fragment, no trailing slash, tracking parameters (utm_*,
fbclid, gclid, ...) removed and the rest sorted.
Types
@type option() :: {:k, non_neg_integer()} | {:engine_boost, number()} | {:lexical_weight, number()} | {:query, String.t()} | {:limit, pos_integer()}
Functions
@spec coverage(Perplexed.Search.Hit.t(), String.t() | MapSet.t()) :: float()
The share of query's terms (lowercased words, stopwords dropped) that
appear in the hit's title, URL or description: 1.0 when the query has no
terms.
@spec fuse([[Perplexed.Search.Hit.t()]], [option()]) :: [Perplexed.Search.Hit.t()]
Fuses lists (each a list of hits, best first) into one ranked list.
Options
:query- the search query, for the lexical factor; without it coverage counts as full.:k- the reciprocal rank constant, default 60.:engine_boost- weight of each extra agreeing engine, default 0.15.:lexical_weight- 0 ignores query terms, 1 scores a hit with none of them at zero; default 0.5.:limit- hits to return, default all.
@spec mean_coverage([Perplexed.Search.Hit.t()], String.t(), pos_integer()) :: float()
The mean coverage/2 of hits' first depth (default 5): how much of
the query a result list talks about. 0.0 for no hits.
The de-duplication key for url: scheme-less, lowercase host without
www., no fragment, no trailing slash, tracking parameters (utm_*,
fbclid, gclid, ...) removed and the rest sorted.
iex> Perplexed.Search.Fusion.url_key("https://www.Example.com/a/?utm_source=x&b=2&a=1#top")
"example.com/a?a=1&b=2"