SpiderMan.Component.ItemProcessor behaviour (spider_man v0.4.1) View Source

Store items.

Life cycle of request:

  1. insert requests to ets of downloader.
  2. downloader's producer get pass out to processes.
  3. processes handle message.
    1. handle by pre pipelines.
    2. call Requester.request/3.
    3. handle by post pipelines.
  4. pass out message.
    1. if success, pass out to Batchers.
    2. if failed, maybe try again.
  5. batcher get enough message and call handle_batch/4.
    1. call Storage.store/3 to save items.
    2. if success, done.
    3. if failed, maybe try again.

Link to this section Summary

Link to this section Callbacks

Link to this callback

process_name(name, base_name)

View Source

Specs

process_name(Broadway.name(), base_name :: String.t()) :: Broadway.name()

Link to this section Functions

Link to this function

process_name(broadway_name, base_name)

View Source