Euros

Euros is a web spider framework that can spider a domain and collect useful information about the pages it visits. It is versatile, allowing you to write your own specialized spider tasks quickly and easily.

Installation

def deps do
  [
    {:euros, "~> 0.2.1"}
  ]
end

Usage

iex> url = "https://euros-test.blogspot.jp/"
iex> Euros.Core.crawl(url, fn(page) -> IO.puts(inspect(page)) end)
%HTTPoison.Response{body: "<!DOCTYPE html><html dir='ltr'><head><meta content='width=device-width, initial-scale=1' name='viewport'/><title>euros test blog</title>....
iex> url = "https://euros-test.blogspot.jp/" 
"https://euros-test.blogspot.jp/"
iex> option = %Euros.HTTPOption{cookie: "foo=bar;"}
%Euros.HTTPOption{cookie: "foo=bar;", recv_timeout: 60000, timeout: 60000}
iex> Euros.Core.crawl(url, fn(page) -> page |> inspect |> IO.puts end, option)

cusotm url pattern

iex> url = "https://euros-test.blogspot.jp/" 
"https://euros-test.blogspot.jp/"
iex> option = %Euros.HTTPOption{cookie: "foo=bar;"}
%Euros.HTTPOption{cookie: "foo=bar;", recv_timeout: 60000, timeout: 60000}
iex> pattern = ~r/test1/
~r/test1/
iex> Euros.Core.crawl(url, fn(page) -> page |> inspect |> IO.puts end, option, pattern)

License

Copyright (c) 2017 kytiken

Released under the MIT license
https://github.com/kytiken/euros/blob/master/LICENSE

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/euros.