View Source API Reference ReqCrawl v0.2.1
Modules
Req plugins to support common crawling functions
A Req plugin to parse robots.txt files
Gathers all URLs from a Sitemap or SitemapIndex according to the specification described at https://sitemaps.org/protocol.html