crawlie v0.5.1 Crawlie.Page

Defines the struct representing a url’s state in the system.

Summary

Types

t()

The Crawlie.Page struct type

Functions

Creates a “child page” - a new Crawlie.Page struct with depth one greate than the one of the parent and no retries

Creates a new Crawlie.Page struct from the url

Returns the Crawlie.Page object with the retry count increased

Returns the string url of the page

Types

t()
t() :: %Crawlie.Page{depth: integer, retries: integer, uri: URI.t}

The Crawlie.Page struct type.

Fields’ meaning:

  • :uri - page URI
  • :depth - the “depth” at which the url was found while recursively crawling the pages. For example depth=0 means it was passed directly from the caller, depth=2 means the crawler followed 2 links from one of the starting urls to get to the url.
  • :retries - url fetch retry count. If the fetching of the url never failed before, 0.

Functions

child(page, uri)

Creates a “child page” - a new Crawlie.Page struct with depth one greate than the one of the parent and no retries.

new(uri, depth \\ 0)
new(URI.t | String.t, integer) :: Crawlie.Page.t

Creates a new Crawlie.Page struct from the url

retry(this)

Returns the Crawlie.Page object with the retry count increased

url(this)

Returns the string url of the page