Kreuzcrawl (kreuzcrawl v0.1.0-rc.9)

High-level API for kreuzcrawl.

Summary

Crawl multiple seed URLs concurrently, each following links to configured depth.

Scrape multiple URLs concurrently.

Method

Crawl a website starting from url, following links up to the configured depth.

Method

Validate the configuration, returning an error if any values are invalid.

Returns the count of unique normalized URLs encountered during crawling.

create_engine()

Create a new crawl engine with the given configuration.

Create a new crawl engine with the given configuration.

Discover all pages on a website by following links and sitemaps.

Scrape a single URL, returning extracted page data.

@spec batch_crawl_async(reference(), [String.t()]) :: [String.t() | nil]

Crawl multiple seed URLs concurrently, each following links to configured depth.

@spec batch_scrape_async(reference(), [String.t()]) :: [String.t() | nil]

Scrape multiple URLs concurrently.

@spec browserconfig_default() :: String.t() | nil

Method

@spec crawl_async(reference(), String.t()) ::
  {:ok, String.t() | nil} | {:error, String.t()}

Crawl a website starting from url, following links up to the configured depth.

@spec crawlconfig_default() :: String.t() | nil

Method

@spec crawlconfig_validate(map()) :: {:ok, nil} | {:error, String.t()}

Validate the configuration, returning an error if any values are invalid.

@spec crawlresult_unique_normalized_urls(map()) :: non_neg_integer()

Returns the count of unique normalized URLs encountered during crawling.

@spec create_engine() :: {:ok, reference()} | {:error, String.t()}

Create a new crawl engine with the given configuration.

@spec create_engine(String.t() | nil | nil) ::
  {:ok, reference()} | {:error, String.t()}

Create a new crawl engine with the given configuration.

@spec map_urls_async(reference(), String.t()) ::
  {:ok, String.t() | nil} | {:error, String.t()}

Discover all pages on a website by following links and sitemaps.

@spec scrape_async(reference(), String.t()) ::
  {:ok, String.t() | nil} | {:error, String.t()}

Scrape a single URL, returning extracted page data.