Gollum.Fetcher
(gollum v0.5.0)
In charge of fetching the actual robots.txt files.
Summary
Fetches the robots.txt file from the specified host. Simply performs
a GET
request to the domain via HTTPoison
.
Functions
Fetches the robots.txt file from the specified host. Simply performs
a GET
request to the domain via HTTPoison
.