URLCategorizationDatabase

Elixir client for content classification of domains and URLs. It is meant to run next to a licensed category file: the file covers known domains offline, and this client classifies whatever the file lacks. Product details and file formats are on the URL categorization database pricing page.

Installation

{:urlcategorizationdatabase, "~> 1.0"}

Classify

client = URLCategorizationDatabase.Client.new(System.fetch_env!("AQ_API_KEY"))
{:ok, body} = URLCategorizationDatabase.Client.classify(client, "arstechnica.com")

classify/2 posts query, data_type=url and the key as a form to the classification endpoint. body is the decoded JSON map. Its fields follow the API reference on the website, since the client passes them through unchanged.

The endpoint reports problems in the body as well as through the status, for example a "status" of 401 together with a message when the key is missing. Check both:

case URLCategorizationDatabase.Client.classify(client, host) do
  {:ok, %{"status" => s} = b} when is_integer(s) and s >= 400 -> {:error, {:api_error, s, b}}
  {:ok, %{"detail" => _} = b} -> {:error, {:api_error, :unknown, b}}
  other -> other
end

An enrichment pipeline with Oban

Most real work is a table of domains that need a category column. Oban turns that into durable, retryable jobs:

defmodule MyApp.Workers.Categorize do
  use Oban.Worker, queue: :categorize, max_attempts: 5

  @impl true
  def perform(%Oban.Job{args: %{"domain" => domain}}) do
    client = URLCategorizationDatabase.Client.new(Application.fetch_env!(:my_app, :ucd_key))

    case URLCategorizationDatabase.Client.classify(client, domain) do
      {:ok, body} ->
        MyApp.Sites.put_categories(domain, body)

      {:error, {:api_error, 429, _}} ->
        {:snooze, 60}

      {:error, {:api_error, status, _}} when status in [401, 403] ->
        {:cancel, :key_or_quota}

      {:error, reason} ->
        {:error, reason}
    end
  end
end

Set the queue limit to a small number, for example categorize: 4, to bound concurrency. {:snooze, 60} backs off on rate limits without using up attempts. {:cancel, ...} stops jobs that cannot succeed until a person fixes the key. Oban's own retry handles everything else.

Enqueueing only what is missing

Repo.all(from s in Site, where: is_nil(s.categories), select: s.domain)
|> Enum.map(&MyApp.Workers.Categorize.new(%{"domain" => &1}))
|> Oban.insert_all()

Before enqueueing, remove domains the licensed file already covers. Every call you skip saves quota.

Normalising domains

Lower-case, drop the scheme, path and a leading www., and store that as the key:

def normalise(raw) do
  raw = if String.contains?(raw, "://"), do: raw, else: "https://" <> raw
  raw |> URI.parse() |> Map.fetch!(:host) |> String.downcase() |> String.replace_prefix("www.", "")
end

On analytics and CRM data this often collapses many rows into far fewer unique domains.

Choosing granularity

Pass the registered domain for one label per site. Pass the full host on platforms where subdomains are separate sites. Pass a full URL when individual pages differ, as on large publishers. Mixing forms in one table makes joins unreliable, so pick one per dataset.

Keeping results current

Store the full response body in a :map (jsonb) column with a categorized_at timestamp. Re-enqueue rows older than a few months, oldest first. When a new release of the licensed file arrives, clear live results for domains it now covers, so the file stays the reference and your table holds only the gaps.

Return values

new/2 and classify/2 require non-empty binaries and raise FunctionClauseError otherwise. Requests use Req.post/2, and Req does not retry POST requests by default, so retrying is up to your job runner. Oban is a good fit for that.

Options

new(key, base_url: url) changes the endpoint base, which helps with testing against a local stub. The key travels in the form body over HTTPS, which is what the endpoint expects.

Typical uses

• CRM enrichment: tag company websites with an industry before routing leads.
• Analytics: add a topic dimension to referrer and outbound-link reports.
• Ad operations: screen domain lists before a campaign.
• Security reporting: label traffic by topic, while blocking decisions use K-12 web filtering categories in schools and enterprise categories elsewhere.

AI traffic

General content taxonomies file AI products under software. AI data leakage prevention tools need that AI flag, which takes one more lookup against the AI register. For detecting shadow AI across resolver data, the log audit does the counting.

Other clients: Go, Dart and PHP.

License

MIT. IAB Tech Lab taxonomy names appear for compatibility only.