SpiderMan.Component.Spider (spider_man v0.3.6) View Source

Analyze web pages.

Life cycle of request:

  1. insert responses to ets of spider component.
  2. component's producer get pass out to processes.
  3. processes handle message.
    1. handle by pre pipelines.
    2. call SpiderModule.handle_response/2.
  4. pass out message.
    1. if success, pass out requests to Downloader component's ets.
    2. if success, pass out items to ItemProcessor component's ets.
    3. if failed, maybe try again.

Link to this section Summary

Link to this section Functions