View Source README

MopURL MopURL

MopURL

URL cleaning service. Casts strings to urls, validates hostnames, and removes redundant and tracking elements.

Uses rules from https://github.com/ClearURLs/Rules

usage

Usage

In order to use MopURL, you must start it and provide a :name. For example, you can add this to your supervision tree:

children = [
  {MopURL, name: MyMopURL}
]

Once you started it, you can clean your urls using MopURL.clean(MyMopURL, url).

performance

Performance

Overall: 1000 QPS with local DNS caching server configured (see bench/full_dns.sh) and 50 QPS (see bench/full.sh) without DNS caching

Just URL validation: can handle at least 50k QPS (see bench/url_validation.sh)

Overall performance is rather small because it requires querying and external DNS server. You can speed this up considerably by setting up a local caching DNS server.

For example, using Dnsmasq, Unbound or directly with NetworkManager.

installation

Installation

If available in Hex, the package can be installed by adding mop_url to your list of dependencies in mix.exs:

def deps do
  [
    {:mop_url, "~> 0.1.0"}
  ]
end

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/mop_url.