View Source CommonCrawl (CommonCrawl v0.1.0)
CommonCrawl library helps to interact with Common Crawl data.
Summary
Functions
@spec collinfo() :: [map()]
Cached collinfo from disk.
Examples
CommonCrawl.collinfo()
[%{
"cdx-api" => "https://index.commoncrawl.org/CC-MAIN-2021-43-index",
"id" => "CC-MAIN-2021-43",
"name" => "October 2021 Index",
"timegate" => "https://index.commoncrawl.org/CC-MAIN-2021-43/"
}, ...]
Fetches current collinfo with all available crawls.