SpiderMan behaviour (spider_man v0.3.5) View Source
SpiderMan, a fast high-level web crawling & scraping framework for Elixir.
Components
Each Spider had 3 components, each component has theirs work:
- Downloader: Download request.
- Spider: Analyze web pages.
- ItemProcessor: Store items.
Message flow: Downloader
-> Spider
-> ItemProcessor
.
Spider Life Cycle
Spider.settings()
- Prepare For Start Stage
Spider.prepare_for_start(:pre, state)
Spider.prepare_for_start_component(:downloader, state)
Spider.prepare_for_start_component(:spider, state)
Spider.prepare_for_start_component(:item_processor, state)
Spider.prepare_for_start(:post, state)
Spider.init(state)
Spider.handle_response(response, context)
- Prepare For Stop Stage
Spider.prepare_for_stop_component(:downloader, state)
Spider.prepare_for_stop_component(:spider, state)
Spider.prepare_for_stop_component(:item_processor, state)
Spider.prepare_for_stop(state)
Link to this section Summary
Types
:print_stats
- Print the stats of spider, The default value istrue
.
Functions
continue a spider
fetch spider's statistics of all ets
fetch spider's state
insert a request to spider
insert multiple requests to spider
list spiders where already started
retry failed events for a spider
start a spider
fetch spider's statistics
fetch component's statistics
fetch spider's status
stop a spider
suspend a spider
Link to this section Types
Specs
component() :: :downloader | :spider | :item_processor
Specs
ets_stats() :: [size: pos_integer(), memory: pos_integer()] | nil
Specs
prepare_for_start_stage() :: :pre | :post
Specs
request() :: SpiderMan.Request.t()
Specs
requests() :: [request()]
Specs
settings() :: keyword()
:print_stats
- Print the stats of spider, The default value istrue
.:log2file
- Save the log to files, The default value istrue
.:status
- Set the startup status for the spider, The default value is:running
.:spider_module
- Set the callback module for the spider,:ets_file
- Set the filename for the spider, and load spider's state from ets files.:downloader_options
- see Downloader Options.:spider_options
- see Spider Options.:item_processor_options
- see ItemProcessor Options.
Downloader options
:requester
- The default value is{{SpiderMan.Requester.Finch, []}}
.:producer
- The default value isSpiderMan.Producer.ETS
.:context
- The default value is%{}
.:processor
- See Processors Options, The default value is[max_demand: 1]
.:concurrency
- The default value is8
.:min_demand
:max_demand
- The default value is10
.:partition_by
:spawn_opt
:hibernate_after
:rate_limiting
- See Producers Options - rate_limiting, The default value is[allowed_messages: 10, interval: 1000]
.:allowed_messages
- Required.:interval
- Required.
:pipelines
- Each msg will handle by each pipelines, The default value is[SpiderMan.Pipeline.DuplicateFilter]
.:post_pipelines
- Each msg will handle by each pipelines, The default value is[]
.
Spider options
:producer
- The default value isSpiderMan.Producer.ETS
.:context
- The default value is%{}
.:processor
- See Processors Options, The default value is[max_demand: 1]
.:concurrency
- The default value is8
.:min_demand
:max_demand
- The default value is10
.:partition_by
:spawn_opt
:hibernate_after
:rate_limiting
- See Producers Options - rate_limiting,:allowed_messages
- Required.:interval
- Required.
:pipelines
- Each msg will handle by each pipelines, The default value is[]
.
Batchers options
:concurrency
- The default value is1
.:batch_size
- The default value is100
.:batch_timeout
- The default value is1000
.:partition_by
:spawn_opt
:hibernate_after
ItemProcessor options
:storage
- Set a storage module what are store items, The default value isSpiderMan.Storage.JsonLines
.:batchers
- See Batchers Options, The default value is[default: [concurrency: 1, batch_size: 50, batch_timeout: 1000]]
.:producer
- The default value isSpiderMan.Producer.ETS
.:context
- The default value is%{}
.:processor
- See Processors Options, The default value is[]
.:concurrency
- The default value is8
.:min_demand
:max_demand
- The default value is10
.:partition_by
:spawn_opt
:hibernate_after
:rate_limiting
- See Producers Options - rate_limiting,:allowed_messages
- Required.:interval
- Required.
:pipelines
- Each msg will handle by each pipelines, The default value is[SpiderMan.Pipeline.DuplicateFilter]
.
Specs
Specs
status() :: :running | :suspended
Link to this section Callbacks
Specs
handle_response(SpiderMan.Response.t(), context :: map()) :: %{ optional(:requests) => [SpiderMan.Request.t()], optional(:items) => [SpiderMan.Item.t()] }
Specs
init(state) :: state when state: SpiderMan.Engine.state()
Specs
prepare_for_start(prepare_for_start_stage(), state) :: state when state: SpiderMan.Engine.state()
Specs
Specs
prepare_for_stop(SpiderMan.Engine.state()) :: :ok
Specs
Specs
settings() :: settings()
Link to this section Functions
Specs
continue a spider
Specs
ets_stats(spider()) :: [ common_pipeline_tid: ets_stats(), downloader_tid: ets_stats(), failed_tid: ets_stats(), spider_tid: ets_stats(), item_processor_tid: ets_stats() ]
fetch spider's statistics of all ets
Specs
get_state(spider()) :: SpiderMan.Engine.state()
fetch spider's state
Specs
insert a request to spider
Specs
insert multiple requests to spider
Specs
list_spiders() :: [spider()]
list spiders where already started
Specs
retry failed events for a spider
Specs
Specs
Specs
start(spider(), settings()) :: Supervisor.on_start_child()
start a spider
Specs
stats(spider()) :: [ status: status(), common_pipeline_tid: ets_stats(), downloader_tid: ets_stats(), failed_tid: ets_stats(), spider_tid: ets_stats(), item_processor_tid: ets_stats() ]
fetch spider's statistics
Specs
fetch component's statistics
Specs
fetch spider's status
Specs
stop(spider()) :: :ok | {:error, error} when error: :not_found | :running | :restarting
stop a spider
Specs
suspend a spider