Crawler v0.3.1 Crawler.Store

An internal data store for information related to each crawl.

Link to this section Summary

Functions

Adds a URL to the registry

Adds the page data for a URL to the registry

Finds a stored URL and returns its page data

Finds a stored URL and returns its page data only if it’s processed

Initialises a new Registry named Crawler.Store.DB

Marks a URL as processed in the registry

Link to this section Functions

Adds a URL to the registry.

Link to this function add_page_data(url, body, opts)

Adds the page data for a URL to the registry.

Finds a stored URL and returns its page data.

Link to this function find_processed(url)

Finds a stored URL and returns its page data only if it’s processed.

Initialises a new Registry named Crawler.Store.DB.

Marks a URL as processed in the registry.