Keyless web search through SearXNG metasearch instances, as a
Perplexed.Search provider (:searxng). Free: price 0.
You give it the instances (searxng_instances: in the settings, or
:instances per call); Perplexed.Search.SearXNG.Directory can pick
healthy public ones from searx.space, filtered by country. Public
instances are volunteers' servers: most close the JSON API and run a bot
limiter, so this sends one HTML search at a time and backs off from any
instance that refuses. Prefer your own instances, or ones whose operators
welcome the traffic, for anything beyond experiments.
The waterfall
Perplexed.Search.SearXNG.Pool ranks the instances by what this node has
measured: response-time and failure averages, plus a cooldown after a
refusal. A search asks the best :merge instances concurrently; each one
that fails (refused, challenged, timed out, empty) is replaced by the next
in the ranking, until :merge have answered or :attempts instances
have been tried. Every outcome feeds back into the pool, so the ranking
follows the instances as they speed up, slow down or start refusing.
With merge: 1 (the default) the hits are the one instance's, re-ranked;
with more, the instances' lists are fused by Perplexed.Search.Fusion
(reciprocal rank, engine agreement, query-term coverage).
Options
:instances- base URLs, or:autofor the live public instancesPerplexed.Search.SearXNG.Instancesdiscovers and keeps fresh; default thesearxng_instancessetting.:merge- instances to combine, default 1.:attempts- instances to try at most, default 4.:timeout_ms- per-instance wait, default 6000.:limit- hits to return, default 8.:format-:html(default) or:jsonfor instances with the JSON API open.:language(default"en-US"),:categories(default"general"),:time_range("day","month","year") and:safesearch(0-2).:proxy-"http://user:pass@host:port", sent through CloakedReq's proxy support, for a residential or rotating proxy.:impersonate- the CloakedReq browser profile: a profile atom, a list to pick from at random per request, or:random(the default), which picks from a set of current Chrome, Edge, Firefox and Safari desktop profiles. A fresh fingerprint per request spreads us across what fingerprint-based limiters see; it does nothing against limits on the IP, which is what refused us most in testing.:min_coverage- lowest average share of query terms in an instance's top 5 hits, default 0.4; below it the answer counts as:irrelevantand the waterfall moves on. Instances pass on results from engines that localize to the server's country or answer bots with unrelated pages, and fusion can only reorder a list, not rescue one that is all off-topic. 0 turns the gate off.:fusion- options forPerplexed.Search.Fusion.fuse/2.:jitter- random spread on each instance's score, 0 to 1, default 0.25, so searches share the fastest few instances instead of piling onto one.:min_interval_ms- an instance used this recently is tried only after every rested one, default 15000. Public instances refuse a client that searches them again too soon (SearXNG's limiter redirects it to the start page), so spreading the load is also what keeps them answering.:pool- thePoolto rank with, default the application's.
Every instance URL passes Perplexed.UrlGuard before each search (DNS can
change after an instance is listed), so an instance on a private network
is refused as :unsafe_url; pass url_validator: to allow a self-hosted
one there.
Each attempt emits [:perplexed, :searxng, :attempt] with duration
(ms) and hits, and metadata instance, outcome and retry_after_ms
(nil unless the instance sent a retry header).
Summary
Functions
Runs the waterfall without fusing: {:ok, {answers, failures}}, where
answers are {instance, hits} in the order the instances answered and
failures are {instance, reason}. For custom fusion or experiments;
search/2 is collect/2 followed by Perplexed.Search.Fusion.fuse/2.
Functions
@spec collect(String.t(), keyword()) :: {:ok, {[{String.t(), [Perplexed.Search.Hit.t()]}], [{String.t(), atom()}]}} | {:error, Perplexed.error()}
Runs the waterfall without fusing: {:ok, {answers, failures}}, where
answers are {instance, hits} in the order the instances answered and
failures are {instance, reason}. For custom fusion or experiments;
search/2 is collect/2 followed by Perplexed.Search.Fusion.fuse/2.