View Source CommonCrawl (CommonCrawl v0.1.0)

CommonCrawl library helps to interact with Common Crawl data.

Summary

Functions

Cached collinfo from disk.

Fetches current collinfo with all available crawls.

Functions

@spec collinfo() :: [map()]

Cached collinfo from disk.

Examples

CommonCrawl.collinfo()
[%{
  "cdx-api" => "https://index.commoncrawl.org/CC-MAIN-2021-43-index",
  "id" => "CC-MAIN-2021-43",
  "name" => "October 2021 Index",
  "timegate" => "https://index.commoncrawl.org/CC-MAIN-2021-43/"
}, ...]
Link to this function

get_collinfo(opts \\ [])

View Source

Fetches current collinfo with all available crawls.