Crawler v0.3.1 Crawler.Store
An internal data store for information related to each crawl.
Link to this section Summary
Functions
Adds a URL to the registry
Adds the page data for a URL to the registry
Finds a stored URL and returns its page data
Finds a stored URL and returns its page data only if it’s processed
Marks a URL as processed in the registry
Link to this section Functions
Link to this function
add(url)
Adds a URL to the registry.
Link to this function
add_page_data(url, body, opts)
Adds the page data for a URL to the registry.
Link to this function
find(url)
Finds a stored URL and returns its page data.
Link to this function
find_processed(url)
Finds a stored URL and returns its page data only if it’s processed.
Link to this function
init()
Initialises a new Registry
named Crawler.Store.DB
.
Link to this function
processed(url)
Marks a URL as processed in the registry.