View Source SpiderMan behaviour (spider_man v0.4.6)

SpiderMan, a fast high-level web crawling & scraping framework for Elixir.

components

Components

Each Spider had 3 components, each component has theirs work:

Message flow: Downloader -> Spider -> ItemProcessor.

spider-life-cycle

Spider Life Cycle

  1. Spider.settings()
  2. Prepare For Start Stage
    1. Spider.prepare_for_start(:pre, state)
    2. Spider.prepare_for_start_component(:downloader, state)
    3. Spider.prepare_for_start_component(:spider, state)
    4. Spider.prepare_for_start_component(:item_processor, state)
    5. Spider.prepare_for_start(:post, state)
  3. Spider.init(state)
  4. Spider.handle_response(response, context)
  5. Prepare For Stop Stage
    1. Spider.prepare_for_stop_component(:downloader, state)
    2. Spider.prepare_for_stop_component(:spider, state)
    3. Spider.prepare_for_stop_component(:item_processor, state)
    4. Spider.prepare_for_stop(state)

Link to this section Summary

Types

  • :print_stats (boolean/0) - Print the stats of spider, The default value is true.

Functions

fetch spider's statistics of all ets

fetch spider's state

insert a request to spider

insert multiple requests to spider

list spiders where already started

fetch spider's statistics

fetch component's statistics

fetch spider's status

stop a spider

Link to this section Types

@type component() :: :downloader | :spider | :item_processor
@type ets_stats() :: [size: pos_integer(), memory: pos_integer()] | nil
Link to this type

prepare_for_start_stage()

View Source
@type prepare_for_start_stage() :: :pre | :post
@type request() :: SpiderMan.Request.t()
@type requests() :: [request()]
@type settings() :: keyword()
  • :print_stats (boolean/0) - Print the stats of spider, The default value is true.

  • :log2file - Save the log to files, The default value is true.

  • :status - Set the startup status for the spider, The default value is :running.

  • :spider_module (atom/0) - Set the callback module for the spider,

  • :ets_file (String.t/0) - Set the filename for the spider, and load spider's state from ets files.

  • :downloader_options (keyword/0) - see Downloader Options.

  • :spider_options (keyword/0) - see Spider Options.

  • :item_processor_options (keyword/0) - see ItemProcessor Options.

downloader-options

Downloader options

spider-options

Spider options

Batchers options

itemprocessor-options

ItemProcessor options

@type spider() :: module() | atom()
@type status() :: :running | :suspended

Link to this section Callbacks

Link to this callback

handle_response(t, context)

View Source
@callback handle_response(SpiderMan.Response.t(), context :: map()) :: %{
  optional(:requests) => [SpiderMan.Request.t()],
  optional(:items) => [SpiderMan.Item.t()]
}
@callback init(state) :: state when state: SpiderMan.Engine.state()
Link to this callback

prepare_for_start(prepare_for_start_stage, state)

View Source (optional)
@callback prepare_for_start(prepare_for_start_stage(), state) :: state
when state: SpiderMan.Engine.state()
Link to this callback

prepare_for_start_component(component, arg2)

View Source (optional)
@callback prepare_for_start_component(component(), options | false) :: options
when options: keyword()
Link to this callback

prepare_for_stop(state)

View Source (optional)
@callback prepare_for_stop(SpiderMan.Engine.state()) :: :ok
Link to this callback

prepare_for_stop_component(component, options)

View Source (optional)
@callback prepare_for_stop_component(component(), options :: keyword() | false) :: :ok
@callback settings() :: settings()

Link to this section Functions

@spec components() :: [component()]
Link to this function

continue(spider, timeout \\ :infinity)

View Source
@spec continue(spider(), timeout()) :: :ok

continue a spider

@spec ets_stats(spider()) :: [
  common_pipeline_tid: ets_stats(),
  downloader_tid: ets_stats(),
  failed_tid: ets_stats(),
  spider_tid: ets_stats(),
  item_processor_tid: ets_stats()
]

fetch spider's statistics of all ets

@spec get_state(spider()) :: SpiderMan.Engine.state()

fetch spider's state

Link to this function

insert_request(spider, request)

View Source
@spec insert_request(spider(), request()) :: true | nil

insert a request to spider

Link to this function

insert_requests(spider, requests)

View Source
@spec insert_requests(spider(), requests()) :: true | nil

insert multiple requests to spider

@spec list_spiders() :: [spider()]

list spiders where already started

Link to this function

retry_failed(spider, max_retries \\ 3, timeout \\ :infinity)

View Source
@spec retry_failed(spider(), max_retries :: integer(), timeout()) ::
  {:ok, count :: integer()}

retry failed events for a spider

Link to this function

run_until(spider, settings \\ [], fun)

View Source
@spec run_until(spider(), settings(), (... -> any())) :: millisecond :: integer()
Link to this function

run_until_zero(spider, settings \\ [], check_interval \\ 1500)

View Source
@spec run_until_zero(spider(), settings(), check_interval :: integer()) ::
  millisecond :: integer()
Link to this function

start(spider, settings \\ [])

View Source
@spec start(spider(), settings()) :: Supervisor.on_start_child()

start a spider

@spec stats(spider()) :: [
  status: status(),
  common_pipeline_tid: ets_stats(),
  downloader_tid: ets_stats(),
  failed_tid: ets_stats(),
  spider_tid: ets_stats(),
  item_processor_tid: ets_stats()
]

fetch spider's statistics

Link to this function

stats(spider, component)

View Source
@spec stats(spider(), component()) :: ets_stats()

fetch component's statistics

@spec status(spider()) :: status()

fetch spider's status

@spec stop(spider()) :: :ok | {:error, error}
when error: :not_found | :running | :restarting

stop a spider

Link to this function

suspend(spider, timeout \\ :infinity)

View Source
@spec suspend(spider(), timeout()) :: :ok

suspend a spider