SpiderMan.Component.Spider behaviour (spider_man v0.4.1) View Source
Analyze web pages.
Life cycle of request:
- insert responses to ets of spider component.
- component's producer get pass out to processes.
- processes handle message.
- handle by pre pipelines.
- call
SpiderModule.handle_response/2
.
- pass out message.
- if success, pass out requests to Downloader component's ets.
- if success, pass out items to ItemProcessor component's ets.
- if failed, maybe try again.
Link to this section Summary
Link to this section Callbacks
Specs
process_name(Broadway.name(), base_name :: String.t()) :: Broadway.name()