View Source ReqCrawl (ReqCrawl v0.2.0)
Req plugins to support common crawling functions
Currently supports parsing the following:
robots.txt
sitemap.{xml,txt}
Summary
Functions
Attach all plugins
Req plugins to support common crawling functions
Currently supports parsing the following:
robots.txt
sitemap.{xml,txt}
Attach all plugins