crawlie v0.1.1 Crawlie

Summary

Functions

Crawls the urls provided in source, using the Crawlie.ParserLogic provided in parser_logic

Functions

crawl(source, parser_logic, options \\ [])
crawl(Stream.t, module, Keyword.t) :: Stream.t

Crawls the urls provided in source, using the Crawlie.ParserLogic provided in parser_logic.

The options are used to tweak the crawler’s behaviour. You can use most of the options for HttPoison, as well as Crawlie specific options.

Crawlie options

  • :http_client - module implementing the Crawlie.HttpClient behaviour to be used to make the requests.
  • :mock_client_fun - If you’re using the Crawlie.HttpClient.MockClient, this would be the url -> {:ok, body :: String.t} | {:error, term} function simulating making the requests.