Crawly v0.7.0 Crawly.Middlewares.DomainFilter View Source
Filters out requests which are going outside of the crawled domain.
The domain that is used to compare against the request url is obtained from the spider's c:Crawly.Spider.base_url
callback.
Does not accept any options. Tuple-based configuration optionswill be ignored.
Example Declaration
middlewares: [
Crawly.Middlewares.DomainFilter
]
Link to this section Summary
Functions
Callback implementation for Crawly.Pipeline.run/3
.
Link to this section Functions
Link to this function
run(request, state, opts \\ []) View Source
Callback implementation for Crawly.Pipeline.run/3
.