View Source README
MopURL
URL cleaning service. Casts strings to urls, validates hostnames, and removes redundant and tracking elements.
Uses rules from https://github.com/ClearURLs/Rules
usage
Usage
In order to use MopURL, you must start it and provide a :name
.
For example, you can add this to your supervision tree:
children = [
{MopURL, name: MyMopURL}
]
Once you started it, you can clean your urls using MopURL.clean(MyMopURL, url)
.
performance
Performance
Overall: 1000 QPS with local DNS caching server configured (see bench/full_dns.sh
)
and 50 QPS (see bench/full.sh
) without DNS caching
Just URL validation: can handle at least 50k QPS
(see bench/url_validation.sh
)
Overall performance is rather small because it requires querying and external DNS server. You can speed this up considerably by setting up a local caching DNS server.
For example, using Dnsmasq, Unbound or directly with NetworkManager.
installation
Installation
If available in Hex, the package can be installed
by adding mop_url
to your list of dependencies in mix.exs
:
def deps do
[
{:mop_url, "~> 0.1.0"}
]
end
Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/mop_url.