gollum v0.2.2 Gollum.Fetcher

In charge of fetching the actual robots.txt files.

Link to this section Summary

Functions

Fetches the robots.txt file from the specified host. Simply performs a GET request to the domain via HTTPoison

Link to this section Functions

Link to this function fetch(domain, opts)
fetch(binary, keyword) :: {:ok, binary} | {:error, term}

Fetches the robots.txt file from the specified host. Simply performs a GET request to the domain via HTTPoison.