crawlie v0.5.1 Crawlie.Page
Defines the struct representing a url’s state in the system.
Summary
Types
The Crawlie.Page
struct type
Functions
Creates a “child page” - a new Crawlie.Page
struct with depth one greate than
the one of the parent and no retries
Creates a new Crawlie.Page
struct from the url
Returns the Crawlie.Page
object with the retry count increased
Returns the string url of the page
Types
The Crawlie.Page
struct type.
Fields’ meaning:
:uri
- pageURI
:depth
- the “depth” at which the url was found while recursively crawling the pages. For exampledepth=0
means it was passed directly from the caller,depth=2
means the crawler followed 2 links from one of the starting urls to get to the url.:retries
- url fetch retry count. If the fetching of the url never failed before,0
.
Functions
Creates a “child page” - a new Crawlie.Page
struct with depth one greate than
the one of the parent and no retries.
Creates a new Crawlie.Page
struct from the url
Returns the Crawlie.Page
object with the retry count increased
Returns the string url of the page