View Source API Reference ReqCrawl v0.2.1

Modules

Req plugins to support common crawling functions

A Req plugin to parse robots.txt files

Gathers all URLs from a Sitemap or SitemapIndex according to the specification described at https://sitemaps.org/protocol.html