API Reference gollum v0.5.0

Modules

Robots.txt parser with caching. Modelled after Kryten.

Caches the robots.txt files from different hosts in memory.

In charge of fetching the actual robots.txt files.

Represents one host's robots.txt files.

Parses a robots.txt file.