FeedReader (FeedReader v0.2.0)
Copy MarkdownStandalone feed fetching and parsing library.
This module exposes format detection, validation, local parsing, and streaming helpers. Remote sessions download and checksum into a temporary file before exposing their one-pass event stream.
Application configuration
Configure stable defaults under :feed_reader, :options. Per-call options
override configured values, which override the defaults below. The application
value must be a keyword list; other values are ignored. :feed_url,
:previous_etag, :previous_checksum, and :item_filter are request-specific
and can only be supplied to individual calls. Options owned by custom formats
or transports can also be configured and are passed to those integrations.
config :feed_reader, :options,
max_items: 1_000,
max_xml_depth: 128,
formats: [MyApp.NoticeFormat],
http_client: MyApp.FeedTransport:formats adds custom format modules after the built-ins. :http_client
selects a custom transport, which receives its own transport-specific options
along with the full option list. For Req request options, configure
:http_options instead.
Option defaults
These defaults apply unless overridden per call or in application config. Resource limits bound retained data and parsing work to avoid memory or CPU spikes and make resource use more predictable across feeds. Collection options without a limit collect all accepted items. Req and custom integration options are defined by their respective integrations.
| Option | Default |
|---|---|
:format | Automatic detection |
:formats | [] additional formats; RSS, Atom, and RDF are always tried first |
:max_items | Unset by default (collect all); a positive value stops parsing after that many accepted items |
:item_filter | Accept all items; per-call only |
:max_detection_bytes | 65_536 bytes |
:chunk_size | 65_536 bytes |
:max_xml_events | 1_000_000 |
:max_xml_depth | 256 |
:max_attributes_per_element | 128 |
:max_attribute_bytes | 65_536 bytes |
:max_feed_metadata_bytes | 1_048_576 bytes |
:max_item_bytes | 4_194_304 bytes |
:max_queue_bytes | :max_item_bytes |
:max_parse_passes | 5 (verified APIs) |
:http_client | FeedReader.ReqTransport (remote APIs) |
:http_options | []; default transport supplies headers and streaming settings |
:http_retry_attempts | 5 (remote APIs) |
:retry_delay | 0 milliseconds (remote APIs) |
:max_compressed_bytes | 52_428_800 bytes (remote APIs) |
:max_decoded_bytes | 52_428_800 bytes (remote APIs) |
:max_temp_bytes | 52_428_800 bytes (remote APIs) |
:max_expansion_ratio | 100 (remote APIs) |
:feed_url | nil locally; final response URL remotely (per-call only) |
:previous_etag | nil (remote, per-call only) |
:previous_checksum | nil (remote, per-call only) |
Resource-limit validation, units, and behavior are detailed in the
Resource limits guide. See each function's options section for option-specific
behavior.
Summary
Types
:rss, :atom, :rdf, :unknown, or a custom format's atom ID.
Functions
Closes an owned remote session; safe after completion or repeated calls.
Detects a format from a bounded binary prefix.
Downloads and collects a feed. :max_items may be supplied to stop parsing
after collecting that many accepted items.
Collects XML binary content into a parse result.
Collects a caller-owned local XML file into a parse result.
Collects an enumerable of binary XML chunks into a parse result.
Downloads and checksums a feed before returning a lazy event session.
Lazily parses a caller-owned local XML file in one SAX pass.
Validates the complete caller-owned local feed before returning an event session.
Downloads and validates a feed before exposing its remote event session.
Checks only the bounded-prefix signature for a format ID or module.
Types
@type format_id() :: atom()
:rss, :atom, :rdf, :unknown, or a custom format's atom ID.
Functions
@spec close(FeedReader.Session.t()) :: :ok
Closes an owned remote session; safe after completion or repeated calls.
Examples
{:ok, session} = FeedReader.stream("https://example.com/feed.xml")
try do
Enum.take(session.events, 10)
after
FeedReader.close(session)
endSee guides/streaming.md for remote-session ownership and cleanup.
Detects a format from a bounded binary prefix.
Examples
xml = ~s(<rss version="2.0"><channel/></rss>)
FeedReader.detect_format(xml)
#=> :rssSee guides/customization.md for custom format modules.
Options
:max_detection_bytes— positive byte limit for the inspected prefix; defaults to65536. Invalid values yield:unknown.:formats— custom format modules checked after RSS, Atom, and RDF in list order. Duplicates are removed. Modules providematches_prefix?/1andid/0.
@spec fetch_and_parse(String.t(), keyword()) :: {:ok, FeedReader.RemoteResult.t() | :not_changed | :not_modified} | {:error, term()}
Downloads and collects a feed. :max_items may be supplied to stop parsing
after collecting that many accepted items.
The default HTTP transport follows redirects automatically, up to five. The
final response URL is available in result.fetch.final_url. Custom transports
control redirect handling and must report the final URL.
Memory use is proportional to the number and size of collected items. Without
:max_items, all accepted items are collected. When the limit is reached, the
result is returned with item_limit_reached?: true; the parser stops there,
so later feed content is not validated. The full remote response is downloaded
before parsing begins.
Examples
{:ok, result} =
FeedReader.fetch_and_parse("https://example.com/feed.xml")
result.feed.title
result.fetch.etagOptions
:http_client—FeedReader.Transportimplementation; defaults toFeedReader.ReqTransport.:http_options— Req options forwarded toReq.run/2. The default transport forces raw streaming and disables automatic decompression.:headersinside this list replaces the default user-agent and accepted-compression headers.:previous_etag— binary ETag sent verbatim asIf-None-Matchon GET; a 304 returns:not_modifiedwithout downloading a body.:http_retry_attempts— non-negative retries after the initial GET; defaults to5. The Req transport retries status 400 and 500 and most transport errors; TLS and redirect failures are terminal. Redirects are limited to five.:retry_delay— non-negative milliseconds or an arity-one delay function; defaults to0. Invalid values use the default.:previous_checksum— decoded-body checksum; a match returns:not_changedand removes the downloaded file.:max_compressed_bytes,:max_decoded_bytes, and:max_temp_bytes— non-negative raw-body, decoded-body, and temporary-file byte limits; defaults are52428800bytes each.:max_expansion_ratio— non-negative decoded-to-compressed byte ratio; defaults to100. Invalid limits use their defaults. Checksums are computed over decoded bytes.:format— format ID orFeedReader.Formatmodule; defaults to detection.:formats— additional format modules, checked after RSS, Atom, and RDF.:chunk_size— positive XML input chunk size; defaults to65536bytes.:max_detection_bytes— positive detection prefix limit; defaults to65536bytes. Invalid values disable detection, so provide a valid explicit:formatto parse in that case.:feed_url— source URL supplied to normalization; defaults tonilfor local sources and the final response URL for remote sources.:max_xml_events,:max_xml_depth,:max_attributes_per_element,:max_attribute_bytes,:max_feed_metadata_bytes, and:max_item_bytes— non-negative SAX and retained-data limits. Defaults are respectively1000000,256,128,65536,1048576, and4194304. Zero is valid; invalid values use the defaults.:max_queue_bytes— non-negative maximum serialized size of one pending event; defaults to:max_item_bytes. Zero is valid; invalid values use that default.Verified APIs also accept
:max_parse_passes(default5), counting validation and delivery.:max_items— optional positive limit for accepted items. When reached, parsing stops and the result'sitem_limit_reached?field istrue; the result may contain only a prefix of the feed. If omitted, all accepted items are collected.:item_filter— optional one-argument function applied before counting and retaining items; invalid values return{:error, {:invalid_option, :item_filter}}.
@spec parse_binary(binary(), keyword()) :: {:ok, FeedReader.ParseResult.t()} | {:error, term()}
Collects XML binary content into a parse result.
:max_items is an optional positive limit; when reached, parsing stops and
item_limit_reached? is set to true. Without it, all accepted items are
collected.
Examples
xml = File.read!("path/to/feed.xml")
{:ok, result} = FeedReader.parse_binary(xml)
result.feed.titleOptions
:format— format ID orFeedReader.Formatmodule; defaults to detection.:formats— additional format modules, checked after RSS, Atom, and RDF.:chunk_size— positive XML input chunk size; defaults to65536bytes.:max_detection_bytes— positive detection prefix limit; defaults to65536bytes. Invalid values disable detection, so provide a valid explicit:formatto parse in that case.:feed_url— source URL supplied to normalization; defaults tonilfor local sources and the final response URL for remote sources.:max_xml_events,:max_xml_depth,:max_attributes_per_element,:max_attribute_bytes,:max_feed_metadata_bytes, and:max_item_bytes— non-negative SAX and retained-data limits. Defaults are respectively1000000,256,128,65536,1048576, and4194304. Zero is valid; invalid values use the defaults.:max_queue_bytes— non-negative maximum serialized size of one pending event; defaults to:max_item_bytes. Zero is valid; invalid values use that default.Verified APIs also accept
:max_parse_passes(default5), counting validation and delivery.:max_items— optional positive limit for accepted items. When reached, parsing stops and the result'sitem_limit_reached?field istrue; the result may contain only a prefix of the feed. If omitted, all accepted items are collected.:item_filter— optional one-argument function applied before counting and retaining items; invalid values return{:error, {:invalid_option, :item_filter}}.
@spec parse_file(Path.t(), keyword()) :: {:ok, FeedReader.ParseResult.t()} | {:error, term()}
Collects a caller-owned local XML file into a parse result.
If :max_items is supplied, parsing stops as soon as that many accepted items
have been collected. The result's item_limit_reached? field is true, and
the rest of the feed is not validated. Without the option, all accepted items
are collected. The parser stops after the feed's root element, ignoring
content appended after a complete feed.
Examples
{:ok, result} = FeedReader.parse_file("path/to/feed.xml")
Enum.map(result.items, & &1.title)Options
:format— format ID orFeedReader.Formatmodule; defaults to detection.:formats— additional format modules, checked after RSS, Atom, and RDF.:chunk_size— positive XML input chunk size; defaults to65536bytes.:max_detection_bytes— positive detection prefix limit; defaults to65536bytes. Invalid values disable detection, so provide a valid explicit:formatto parse in that case.:feed_url— source URL supplied to normalization; defaults tonilfor local sources and the final response URL for remote sources.:max_xml_events,:max_xml_depth,:max_attributes_per_element,:max_attribute_bytes,:max_feed_metadata_bytes, and:max_item_bytes— non-negative SAX and retained-data limits. Defaults are respectively1000000,256,128,65536,1048576, and4194304. Zero is valid; invalid values use the defaults.:max_queue_bytes— non-negative maximum serialized size of one pending event; defaults to:max_item_bytes. Zero is valid; invalid values use that default.Verified APIs also accept
:max_parse_passes(default5), counting validation and delivery.:max_items— optional positive limit for accepted items. When reached, parsing stops and the result'sitem_limit_reached?field istrue; the result may contain only a prefix of the feed. If omitted, all accepted items are collected.:item_filter— optional one-argument function applied before counting and retaining items; invalid values return{:error, {:invalid_option, :item_filter}}.
@spec parse_stream(Enumerable.t(), keyword()) :: {:ok, FeedReader.ParseResult.t()} | {:error, term()}
Collects an enumerable of binary XML chunks into a parse result.
Chunks are written incrementally to a temporary file and then parsed through
the same event stream used by parse_file/2. The temporary file is removed
on success and failure. :max_items is optional; when reached, parsing stops
and item_limit_reached? is set to true. Input chunks are staged before
parsing begins, so use a streaming API to process large feeds incrementally.
Examples
chunks = File.stream!("path/to/feed.xml", 16_384, [:raw])
{:ok, result} = FeedReader.parse_stream(chunks)
length(result.items)Options
:format— format ID orFeedReader.Formatmodule; defaults to detection.:formats— additional format modules, checked after RSS, Atom, and RDF.:chunk_size— positive XML input chunk size; defaults to65536bytes.:max_detection_bytes— positive detection prefix limit; defaults to65536bytes. Invalid values disable detection, so provide a valid explicit:formatto parse in that case.:feed_url— source URL supplied to normalization; defaults tonilfor local sources and the final response URL for remote sources.:max_xml_events,:max_xml_depth,:max_attributes_per_element,:max_attribute_bytes,:max_feed_metadata_bytes, and:max_item_bytes— non-negative SAX and retained-data limits. Defaults are respectively1000000,256,128,65536,1048576, and4194304. Zero is valid; invalid values use the defaults.:max_queue_bytes— non-negative maximum serialized size of one pending event; defaults to:max_item_bytes. Zero is valid; invalid values use that default.Verified APIs also accept
:max_parse_passes(default5), counting validation and delivery.:max_items— optional positive limit for accepted items. When reached, parsing stops and the result'sitem_limit_reached?field istrue; the result may contain only a prefix of the feed. If omitted, all accepted items are collected.:item_filter— optional one-argument function applied before counting and retaining items; invalid values return{:error, {:invalid_option, :item_filter}}.
@spec stream(String.t(), keyword()) :: {:ok, FeedReader.Session.t() | :not_changed | :not_modified} | {:error, term()}
Downloads and checksums a feed before returning a lazy event session.
The session emits feed metadata followed by items from one local XML pass.
Call close/1 in an after block if enumeration may stop early. Matching
previous checksums or a conditional HTTP 304 return :not_changed or
:not_modified without a session.
Streaming APIs do not enforce :max_items; it applies only to collection APIs.
A stream emits items as the consumer enumerates them. Stop enumeration in the
consumer to limit how many events are processed. Use streaming APIs to process
large feeds incrementally instead of collecting every item in memory.
Examples
url = "https://example.com/feed.xml"
{:ok, session} = FeedReader.stream(url)
try do
session.events
|> Enum.take(20)
|> Enum.each(&IO.inspect/1)
after
FeedReader.close(session)
endSee guides/streaming.md for event handling and remote-session cleanup.
Options
:http_client—FeedReader.Transportimplementation; defaults toFeedReader.ReqTransport.:http_options— Req options forwarded toReq.run/2. The default transport forces raw streaming and disables automatic decompression.:headersinside this list replaces the default user-agent and accepted-compression headers.:previous_etag— binary ETag sent verbatim asIf-None-Matchon GET; a 304 returns:not_modifiedwithout downloading a body.:http_retry_attempts— non-negative retries after the initial GET; defaults to5. The Req transport retries status 400 and 500 and most transport errors; TLS and redirect failures are terminal. Redirects are limited to five.:retry_delay— non-negative milliseconds or an arity-one delay function; defaults to0. Invalid values use the default.:previous_checksum— decoded-body checksum; a match returns:not_changedand removes the downloaded file.:max_compressed_bytes,:max_decoded_bytes, and:max_temp_bytes— non-negative raw-body, decoded-body, and temporary-file byte limits; defaults are52428800bytes each.:max_expansion_ratio— non-negative decoded-to-compressed byte ratio; defaults to100. Invalid limits use their defaults. Checksums are computed over decoded bytes.:format— format ID orFeedReader.Formatmodule; defaults to detection.:formats— additional format modules, checked after RSS, Atom, and RDF.:chunk_size— positive XML input chunk size; defaults to65536bytes.:max_detection_bytes— positive detection prefix limit; defaults to65536bytes. Invalid values disable detection, so provide a valid explicit:formatto parse in that case.:feed_url— source URL supplied to normalization; defaults tonilfor local sources and the final response URL for remote sources.:max_xml_events,:max_xml_depth,:max_attributes_per_element,:max_attribute_bytes,:max_feed_metadata_bytes, and:max_item_bytes— non-negative SAX and retained-data limits. Defaults are respectively1000000,256,128,65536,1048576, and4194304. Zero is valid; invalid values use the defaults.:max_queue_bytes— non-negative maximum serialized size of one pending event; defaults to:max_item_bytes. Zero is valid; invalid values use that default.Verified APIs also accept
:max_parse_passes(default5), counting validation and delivery.
@spec stream_file(Path.t(), keyword()) :: {:ok, FeedReader.Session.t()} | {:error, term()}
Lazily parses a caller-owned local XML file in one SAX pass.
The returned session emits feed metadata before normalized items. The file is never removed by the library, including when enumeration stops early.
Streaming APIs do not enforce :max_items; it applies only to collection APIs.
A stream emits items as the consumer enumerates them. Stop enumeration in the
consumer to limit how many events are processed. Use streaming APIs to process
large feeds incrementally instead of collecting every item in memory.
Examples
{:ok, session} = FeedReader.stream_file("path/to/feed.xml")
Enum.each(session.events, &IO.inspect/1)Options
:format— format ID orFeedReader.Formatmodule; defaults to detection.:formats— additional format modules, checked after RSS, Atom, and RDF.:chunk_size— positive XML input chunk size; defaults to65536bytes.:max_detection_bytes— positive detection prefix limit; defaults to65536bytes. Invalid values disable detection, so provide a valid explicit:formatto parse in that case.:feed_url— source URL supplied to normalization; defaults tonilfor local sources and the final response URL for remote sources.:max_xml_events,:max_xml_depth,:max_attributes_per_element,:max_attribute_bytes,:max_feed_metadata_bytes, and:max_item_bytes— non-negative SAX and retained-data limits. Defaults are respectively1000000,256,128,65536,1048576, and4194304. Zero is valid; invalid values use the defaults.:max_queue_bytes— non-negative maximum serialized size of one pending event; defaults to:max_item_bytes. Zero is valid; invalid values use that default.- Verified APIs also accept
:max_parse_passes(default5), counting validation and delivery.
@spec stream_file_verified(Path.t(), keyword()) :: {:ok, FeedReader.Session.t()} | {:error, term()}
Validates the complete caller-owned local feed before returning an event session.
The source file is never removed. :max_parse_passes bounds validation and
event delivery passes.
Streaming APIs do not enforce :max_items; it applies only to collection APIs.
A stream emits items as the consumer enumerates them. Stop enumeration in the
consumer to limit how many events are processed. Use streaming APIs to process
large feeds incrementally instead of collecting every item in memory.
Examples
{:ok, session} = FeedReader.stream_file_verified("path/to/feed.xml")
Enum.each(session.events, &IO.inspect/1)Options
:format— format ID orFeedReader.Formatmodule; defaults to detection.:formats— additional format modules, checked after RSS, Atom, and RDF.:chunk_size— positive XML input chunk size; defaults to65536bytes.:max_detection_bytes— positive detection prefix limit; defaults to65536bytes. Invalid values disable detection, so provide a valid explicit:formatto parse in that case.:feed_url— source URL supplied to normalization; defaults tonilfor local sources and the final response URL for remote sources.:max_xml_events,:max_xml_depth,:max_attributes_per_element,:max_attribute_bytes,:max_feed_metadata_bytes, and:max_item_bytes— non-negative SAX and retained-data limits. Defaults are respectively1000000,256,128,65536,1048576, and4194304. Zero is valid; invalid values use the defaults.:max_queue_bytes— non-negative maximum serialized size of one pending event; defaults to:max_item_bytes. Zero is valid; invalid values use that default.- Verified APIs also accept
:max_parse_passes(default5), counting validation and delivery.
@spec stream_verified(String.t(), keyword()) :: {:ok, FeedReader.Session.t() | :not_changed | :not_modified} | {:error, term()}
Downloads and validates a feed before exposing its remote event session.
Matching checksums and conditional HTTP 304 responses return :not_changed
and :not_modified without a session. Call close/1 if enumeration may stop
early.
Streaming APIs do not enforce :max_items; it applies only to collection APIs.
A stream emits items as the consumer enumerates them. Stop enumeration in the
consumer to limit how many events are processed. Use streaming APIs to process
large feeds incrementally instead of collecting every item in memory.
Examples
url = "https://example.com/feed.xml"
{:ok, session} = FeedReader.stream_verified(url)
try do
Enum.each(session.events, &IO.inspect/1)
after
FeedReader.close(session)
endSee guides/streaming.md for event handling and remote-session cleanup.
Options
:http_client—FeedReader.Transportimplementation; defaults toFeedReader.ReqTransport.:http_options— Req options forwarded toReq.run/2. The default transport forces raw streaming and disables automatic decompression.:headersinside this list replaces the default user-agent and accepted-compression headers.:previous_etag— binary ETag sent verbatim asIf-None-Matchon GET; a 304 returns:not_modifiedwithout downloading a body.:http_retry_attempts— non-negative retries after the initial GET; defaults to5. The Req transport retries status 400 and 500 and most transport errors; TLS and redirect failures are terminal. Redirects are limited to five.:retry_delay— non-negative milliseconds or an arity-one delay function; defaults to0. Invalid values use the default.:previous_checksum— decoded-body checksum; a match returns:not_changedand removes the downloaded file.:max_compressed_bytes,:max_decoded_bytes, and:max_temp_bytes— non-negative raw-body, decoded-body, and temporary-file byte limits; defaults are52428800bytes each.:max_expansion_ratio— non-negative decoded-to-compressed byte ratio; defaults to100. Invalid limits use their defaults. Checksums are computed over decoded bytes.:format— format ID orFeedReader.Formatmodule; defaults to detection.:formats— additional format modules, checked after RSS, Atom, and RDF.:chunk_size— positive XML input chunk size; defaults to65536bytes.:max_detection_bytes— positive detection prefix limit; defaults to65536bytes. Invalid values disable detection, so provide a valid explicit:formatto parse in that case.:feed_url— source URL supplied to normalization; defaults tonilfor local sources and the final response URL for remote sources.:max_xml_events,:max_xml_depth,:max_attributes_per_element,:max_attribute_bytes,:max_feed_metadata_bytes, and:max_item_bytes— non-negative SAX and retained-data limits. Defaults are respectively1000000,256,128,65536,1048576, and4194304. Zero is valid; invalid values use the defaults.:max_queue_bytes— non-negative maximum serialized size of one pending event; defaults to:max_item_bytes. Zero is valid; invalid values use that default.Verified APIs also accept
:max_parse_passes(default5), counting validation and delivery.
Checks only the bounded-prefix signature for a format ID or module.
Examples
xml = ~s(<rss version="2.0"><channel/></rss>)
FeedReader.validate(xml, :rss)
#=> trueSee guides/customization.md for custom format modules.
Options
:max_detection_bytes— positive byte limit for the inspected prefix; defaults to65536. Invalid values yield:unknown.:formats— custom format modules checked after RSS, Atom, and RDF in list order. Duplicates are removed. Modules providematches_prefix?/1andid/0.