FeedReader (FeedReader v0.4.0)

Copy Markdown

Standalone feed fetching and parsing library.

This module exposes format detection, validation, local parsing, and streaming helpers. Remote sessions download and checksum into a temporary file before exposing their one-pass event stream.

Application configuration

Configure stable defaults under :feed_reader, :options. Per-call options override configured values, which override the defaults below. The application value must be a keyword list; other values are ignored. :feed_url, :previous_etag, :previous_checksum, and :item_filter are request-specific and can only be supplied to individual calls. Options owned by custom formats or transports can also be configured and are passed to those integrations.

config :feed_reader, :options,
  max_items: 1_000,
  max_xml_depth: 128,
  formats: [MyApp.NoticeFormat],
  http_client: MyApp.FeedTransport

:formats adds custom format modules after the built-ins. :http_client selects a custom transport, which receives its own transport-specific options along with the full option list. For Req request options, configure :http_options instead.

Option defaults

These defaults apply unless overridden per call or in application config. Resource limits bound retained data and parsing work to avoid memory or CPU spikes and make resource use more predictable across feeds. Collection options without a limit collect all accepted items. Req and custom integration options are defined by their respective integrations.

OptionDefault
:formatAutomatic detection
:formats[] additional formats; RSS, Atom, and RDF are always tried first
:max_itemsUnset by default (collect all); a positive value stops parsing after that many accepted items
:item_filterAccept all items; per-call only
:max_detection_bytes65_536 bytes
:chunk_size65_536 bytes
:max_xml_events1_000_000
:max_xml_depth256
:max_attributes_per_element128
:max_attribute_bytes65_536 bytes
:max_feed_metadata_bytes1_048_576 bytes
:max_item_bytes4_194_304 bytes
:max_queue_bytes:max_item_bytes
:max_parse_passes5 (verified APIs)
:http_clientFeedReader.ReqTransport (remote APIs)
:http_options[]; default transport supplies headers and streaming settings
:http_retry_attempts5 (remote APIs)
:retry_delay0 milliseconds (remote APIs)
:max_compressed_bytes52_428_800 bytes (remote APIs)
:max_decoded_bytes52_428_800 bytes (remote APIs)
:max_temp_bytes52_428_800 bytes (remote APIs)
:max_expansion_ratio100 (remote APIs)
:feed_urlnil locally; final response URL remotely (per-call only)
:previous_etagnil (remote, per-call only)
:previous_checksumnil (remote, per-call only)

Resource-limit validation, units, and behavior are detailed in the Resource limits guide. See each function's options section for option-specific behavior.

Summary

Types

:rss, :atom, :rdf, :unknown, or a custom format's atom ID.

Functions

Closes an owned remote session; safe after completion or repeated calls.

Detects a format from a bounded binary prefix.

Downloads and collects a feed. :max_items may be supplied to stop parsing after collecting that many accepted items.

Collects XML binary content into a parse result.

Collects a caller-owned local XML file into a parse result.

Collects an enumerable of binary XML chunks into a parse result.

Downloads and checksums a feed before returning a lazy event session.

Lazily parses a caller-owned local XML file in one SAX pass.

Validates the complete caller-owned local feed before returning an event session.

Downloads and validates a feed before exposing its remote event session.

Checks only the bounded-prefix signature for a format ID or module.

Types

format_id()

@type format_id() :: atom()

:rss, :atom, :rdf, :unknown, or a custom format's atom ID.

Functions

close(session)

@spec close(FeedReader.Session.t()) :: :ok

Closes an owned remote session; safe after completion or repeated calls.

Examples

{:ok, session} = FeedReader.stream("https://example.com/feed.xml")

try do
  Enum.take(session.events, 10)
after
  FeedReader.close(session)
end

See guides/streaming.md for remote-session ownership and cleanup.

detect_format(content, opts \\ [])

@spec detect_format(binary(), keyword()) :: format_id()

Detects a format from a bounded binary prefix.

Examples

xml = ~s(<rss version="2.0"><channel/></rss>)
FeedReader.detect_format(xml)
#=> :rss

See guides/customization.md for custom format modules.

Options

  • :max_detection_bytes — positive byte limit for the inspected prefix; defaults to 65536. Invalid values yield :unknown.
  • :formats — custom format modules checked after RSS, Atom, and RDF in list order. Duplicates are removed. Modules provide matches_prefix?/1 and id/0.

fetch_and_parse(url, opts \\ [])

@spec fetch_and_parse(String.t(), keyword()) ::
  {:ok, FeedReader.RemoteResult.t() | :not_changed | :not_modified}
  | {:error, term()}

Downloads and collects a feed. :max_items may be supplied to stop parsing after collecting that many accepted items.

The default HTTP transport follows redirects automatically, up to five. The final response URL is available in result.fetch.final_url. Custom transports control redirect handling and must report the final URL.

Memory use is proportional to the number and size of collected items. Without :max_items, all accepted items are collected. When the limit is reached, the result is returned with item_limit_reached?: true; the parser stops there, so later feed content is not validated. The full remote response is downloaded before parsing begins.

Examples

{:ok, result} =
  FeedReader.fetch_and_parse("https://example.com/feed.xml")

result.feed.title
result.fetch.etag

Options

  • :http_client — FeedReader.Transport implementation; defaults to FeedReader.ReqTransport.

  • :http_options — Req options forwarded to Req.run/2. The default transport forces raw streaming and disables automatic decompression. :headers inside this list replaces the default user-agent and accepted-compression headers.

  • :previous_etag — binary ETag sent verbatim as If-None-Match on GET; a 304 returns :not_modified without downloading a body.

  • :http_retry_attempts — non-negative retries after the initial GET; defaults to 5. The Req transport retries status 400 and 500 and most transport errors; TLS and redirect failures are terminal. Redirects are limited to five.

  • :retry_delay — non-negative milliseconds or an arity-one delay function; defaults to 0. Invalid values use the default.

  • :previous_checksum — decoded-body checksum; a match returns :not_changed and removes the downloaded file.

  • :max_compressed_bytes, :max_decoded_bytes, and :max_temp_bytes — non-negative raw-body, decoded-body, and temporary-file byte limits; defaults are 52428800 bytes each.

  • :max_expansion_ratio — non-negative decoded-to-compressed byte ratio; defaults to 100. Invalid limits use their defaults. Checksums are computed over decoded bytes.

  • :format — format ID or FeedReader.Format module; defaults to detection.

  • :formats — additional format modules, checked after RSS, Atom, and RDF.

  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.

  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.

  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.

  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.

  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.

  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

  • :max_items — optional positive limit for accepted items. When reached, parsing stops and the result's item_limit_reached? field is true; the result may contain only a prefix of the feed. If omitted, all accepted items are collected.

  • :item_filter — optional one-argument function applied before counting and retaining items; invalid values return {:error, {:invalid_option, :item_filter}}.

parse_binary(content, opts \\ [])

@spec parse_binary(binary(), keyword()) ::
  {:ok, FeedReader.ParseResult.t()} | {:error, term()}

Collects XML binary content into a parse result.

:max_items is an optional positive limit; when reached, parsing stops and item_limit_reached? is set to true. Without it, all accepted items are collected.

Examples

xml = File.read!("path/to/feed.xml")
{:ok, result} = FeedReader.parse_binary(xml)
result.feed.title

Options

  • :format — format ID or FeedReader.Format module; defaults to detection.

  • :formats — additional format modules, checked after RSS, Atom, and RDF.

  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.

  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.

  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.

  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.

  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.

  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

  • :max_items — optional positive limit for accepted items. When reached, parsing stops and the result's item_limit_reached? field is true; the result may contain only a prefix of the feed. If omitted, all accepted items are collected.

  • :item_filter — optional one-argument function applied before counting and retaining items; invalid values return {:error, {:invalid_option, :item_filter}}.

parse_file(path, opts \\ [])

@spec parse_file(Path.t(), keyword()) ::
  {:ok, FeedReader.ParseResult.t()} | {:error, term()}

Collects a caller-owned local XML file into a parse result.

If :max_items is supplied, parsing stops as soon as that many accepted items have been collected. The result's item_limit_reached? field is true, and the rest of the feed is not validated. Without the option, all accepted items are collected. The parser stops after the feed's root element, ignoring content appended after a complete feed.

Examples

{:ok, result} = FeedReader.parse_file("path/to/feed.xml")
Enum.map(result.items, & &1.title)

Options

  • :format — format ID or FeedReader.Format module; defaults to detection.

  • :formats — additional format modules, checked after RSS, Atom, and RDF.

  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.

  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.

  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.

  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.

  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.

  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

  • :max_items — optional positive limit for accepted items. When reached, parsing stops and the result's item_limit_reached? field is true; the result may contain only a prefix of the feed. If omitted, all accepted items are collected.

  • :item_filter — optional one-argument function applied before counting and retaining items; invalid values return {:error, {:invalid_option, :item_filter}}.

parse_stream(chunks, opts \\ [])

@spec parse_stream(Enumerable.t(), keyword()) ::
  {:ok, FeedReader.ParseResult.t()} | {:error, term()}

Collects an enumerable of binary XML chunks into a parse result.

Chunks are written incrementally to a temporary file and then parsed through the same event stream used by parse_file/2. The temporary file is removed on success and failure. :max_items is optional; when reached, parsing stops and item_limit_reached? is set to true. Input chunks are staged before parsing begins, so use a streaming API to process large feeds incrementally.

Examples

chunks = File.stream!("path/to/feed.xml", 16_384, [:raw])
{:ok, result} = FeedReader.parse_stream(chunks)
length(result.items)

Options

  • :format — format ID or FeedReader.Format module; defaults to detection.

  • :formats — additional format modules, checked after RSS, Atom, and RDF.

  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.

  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.

  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.

  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.

  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.

  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

  • :max_items — optional positive limit for accepted items. When reached, parsing stops and the result's item_limit_reached? field is true; the result may contain only a prefix of the feed. If omitted, all accepted items are collected.

  • :item_filter — optional one-argument function applied before counting and retaining items; invalid values return {:error, {:invalid_option, :item_filter}}.

stream(url, opts \\ [])

@spec stream(String.t(), keyword()) ::
  {:ok, FeedReader.Session.t() | :not_changed | :not_modified}
  | {:error, term()}

Downloads and checksums a feed before returning a lazy event session.

The session emits feed metadata followed by items from one local XML pass. Call close/1 in an after block if enumeration may stop early. Matching previous checksums or a conditional HTTP 304 return :not_changed or :not_modified without a session.

Streaming APIs do not enforce :max_items; it applies only to collection APIs. A stream emits items as the consumer enumerates them. Stop enumeration in the consumer to limit how many events are processed. Use streaming APIs to process large feeds incrementally instead of collecting every item in memory.

Examples

url = "https://example.com/feed.xml"
{:ok, session} = FeedReader.stream(url)

try do
  session.events
  |> Enum.take(20)
  |> Enum.each(&IO.inspect/1)
after
  FeedReader.close(session)
end

See guides/streaming.md for event handling and remote-session cleanup.

Options

  • :http_client — FeedReader.Transport implementation; defaults to FeedReader.ReqTransport.

  • :http_options — Req options forwarded to Req.run/2. The default transport forces raw streaming and disables automatic decompression. :headers inside this list replaces the default user-agent and accepted-compression headers.

  • :previous_etag — binary ETag sent verbatim as If-None-Match on GET; a 304 returns :not_modified without downloading a body.

  • :http_retry_attempts — non-negative retries after the initial GET; defaults to 5. The Req transport retries status 400 and 500 and most transport errors; TLS and redirect failures are terminal. Redirects are limited to five.

  • :retry_delay — non-negative milliseconds or an arity-one delay function; defaults to 0. Invalid values use the default.

  • :previous_checksum — decoded-body checksum; a match returns :not_changed and removes the downloaded file.

  • :max_compressed_bytes, :max_decoded_bytes, and :max_temp_bytes — non-negative raw-body, decoded-body, and temporary-file byte limits; defaults are 52428800 bytes each.

  • :max_expansion_ratio — non-negative decoded-to-compressed byte ratio; defaults to 100. Invalid limits use their defaults. Checksums are computed over decoded bytes.

  • :format — format ID or FeedReader.Format module; defaults to detection.

  • :formats — additional format modules, checked after RSS, Atom, and RDF.

  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.

  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.

  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.

  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.

  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.

  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

stream_file(path, opts \\ [])

@spec stream_file(Path.t(), keyword()) ::
  {:ok, FeedReader.Session.t()} | {:error, term()}

Lazily parses a caller-owned local XML file in one SAX pass.

The returned session emits feed metadata before normalized items. The file is never removed by the library, including when enumeration stops early.

Streaming APIs do not enforce :max_items; it applies only to collection APIs. A stream emits items as the consumer enumerates them. Stop enumeration in the consumer to limit how many events are processed. Use streaming APIs to process large feeds incrementally instead of collecting every item in memory.

Examples

{:ok, session} = FeedReader.stream_file("path/to/feed.xml")
Enum.each(session.events, &IO.inspect/1)

Options

  • :format — format ID or FeedReader.Format module; defaults to detection.
  • :formats — additional format modules, checked after RSS, Atom, and RDF.
  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.
  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.
  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.
  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.
  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.
  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

stream_file_verified(path, opts \\ [])

@spec stream_file_verified(Path.t(), keyword()) ::
  {:ok, FeedReader.Session.t()} | {:error, term()}

Validates the complete caller-owned local feed before returning an event session.

The source file is never removed. :max_parse_passes bounds validation and event delivery passes.

Streaming APIs do not enforce :max_items; it applies only to collection APIs. A stream emits items as the consumer enumerates them. Stop enumeration in the consumer to limit how many events are processed. Use streaming APIs to process large feeds incrementally instead of collecting every item in memory.

Examples

{:ok, session} = FeedReader.stream_file_verified("path/to/feed.xml")
Enum.each(session.events, &IO.inspect/1)

Options

  • :format — format ID or FeedReader.Format module; defaults to detection.
  • :formats — additional format modules, checked after RSS, Atom, and RDF.
  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.
  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.
  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.
  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.
  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.
  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

stream_verified(url, opts \\ [])

@spec stream_verified(String.t(), keyword()) ::
  {:ok, FeedReader.Session.t() | :not_changed | :not_modified}
  | {:error, term()}

Downloads and validates a feed before exposing its remote event session.

Matching checksums and conditional HTTP 304 responses return :not_changed and :not_modified without a session. Call close/1 if enumeration may stop early.

Streaming APIs do not enforce :max_items; it applies only to collection APIs. A stream emits items as the consumer enumerates them. Stop enumeration in the consumer to limit how many events are processed. Use streaming APIs to process large feeds incrementally instead of collecting every item in memory.

Examples

url = "https://example.com/feed.xml"
{:ok, session} = FeedReader.stream_verified(url)

try do
  Enum.each(session.events, &IO.inspect/1)
after
  FeedReader.close(session)
end

See guides/streaming.md for event handling and remote-session cleanup.

Options

  • :http_client — FeedReader.Transport implementation; defaults to FeedReader.ReqTransport.

  • :http_options — Req options forwarded to Req.run/2. The default transport forces raw streaming and disables automatic decompression. :headers inside this list replaces the default user-agent and accepted-compression headers.

  • :previous_etag — binary ETag sent verbatim as If-None-Match on GET; a 304 returns :not_modified without downloading a body.

  • :http_retry_attempts — non-negative retries after the initial GET; defaults to 5. The Req transport retries status 400 and 500 and most transport errors; TLS and redirect failures are terminal. Redirects are limited to five.

  • :retry_delay — non-negative milliseconds or an arity-one delay function; defaults to 0. Invalid values use the default.

  • :previous_checksum — decoded-body checksum; a match returns :not_changed and removes the downloaded file.

  • :max_compressed_bytes, :max_decoded_bytes, and :max_temp_bytes — non-negative raw-body, decoded-body, and temporary-file byte limits; defaults are 52428800 bytes each.

  • :max_expansion_ratio — non-negative decoded-to-compressed byte ratio; defaults to 100. Invalid limits use their defaults. Checksums are computed over decoded bytes.

  • :format — format ID or FeedReader.Format module; defaults to detection.

  • :formats — additional format modules, checked after RSS, Atom, and RDF.

  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.

  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.

  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.

  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.

  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.

  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

validate(content, format, opts \\ [])

@spec validate(binary(), atom() | module(), keyword()) :: boolean()

Checks only the bounded-prefix signature for a format ID or module.

Examples

xml = ~s(<rss version="2.0"><channel/></rss>)
FeedReader.validate(xml, :rss)
#=> true

See guides/customization.md for custom format modules.

Options

  • :max_detection_bytes — positive byte limit for the inspected prefix; defaults to 65536. Invalid values yield :unknown.
  • :formats — custom format modules checked after RSS, Atom, and RDF in list order. Duplicates are removed. Modules provide matches_prefix?/1 and id/0.