Robots.txt parser with caching. Modelled after Kryten.
Caches the robots.txt files from different hosts in memory.
In charge of fetching the actual robots.txt files.
Represents one host's robots.txt files.
Parses a robots.txt file.