View Source CommonCrawl.WARC (CommonCrawl v0.1.0)

Common Crawl .warc file download and parsing

Summary

Functions

Fetches a segment of the WARC file.

Functions

Link to this function

get_segment(filename, offset, length, opts \\ [])

View Source
@spec get_segment(String.t(), integer(), integer(), keyword()) ::
  {:ok, %{warc: String.t(), headers: String.t(), response: String.t()}}
  | {:error, any()}

Fetches a segment of the WARC file.