Gollum.Fetcher (gollum v0.5.0)

In charge of fetching the actual robots.txt files.

Summary

Functions

Fetches the robots.txt file from the specified host. Simply performs a GET request to the domain via HTTPoison.

Functions

Link to this function

fetch(domain, opts)

@spec fetch(
  binary(),
  keyword()
) :: {:ok, binary()} | {:error, term()}

Fetches the robots.txt file from the specified host. Simply performs a GET request to the domain via HTTPoison.