gollum v0.2.1 Gollum.Fetcher
In charge of fetching the actual robots.txt files.
Link to this section Summary
Link to this section Functions
Fetches the robots.txt file from the specified host. Simply performs
a GET
request to the domain via HTTPoison
.