View Source ReqCrawl (ReqCrawl v0.2.1)

Req plugins to support common crawling functions

Currently supports parsing the following:

  • robots.txt
  • sitemap.{xml,txt}

Summary

Functions

Link to this function

attach_all(request, options \\ [])

View Source

Attach all plugins

Options

Refer to each respective plugin's documentation for their available options.