Crawly v0.2.0 Crawly.Middlewares.UserAgent View Source

Set/Rotate user agents for crawling. The user agents are read from :crawly, :user_agents sessions.

The default value for the user agent is: Crawly Bot 1.0

Link to this section Summary

Link to this section Functions