Streaming and verified parsing

Copy Markdown

Streaming APIs return a %FeedReader.Session{} containing a one-pass events enumerable. Events are delivered in order: one {:feed, feed} event followed by normalized {:item, item} events. A parser error is emitted as a terminal {:error, reason} event. Errors that happen before a session can be created, such as an unknown format or a failed remote request, are returned directly as {:error, reason}.

The feed event includes a single feed-metadata snapshot. RSS and Atom include eligible channel/feed media seen before the first item; RDF uses the channel available when the feed event is emitted. Later feed-level media does not update that event. Item media is emitted with its item in source encounter order. These normalized feed and item values are the same ones returned by collection APIs.

Streaming APIs do not enforce :max_items; collection APIs such as parse_file/2, parse_stream/2, and fetch_and_parse/2 collect all accepted items by default and can use the option as an early-stop cap. A stream emits items as the consumer enumerates them. Stop enumeration in the consumer to limit how many events are processed.

Stream a local file

{:ok, session} = FeedReader.stream_file("path/to/feed.xml")

session.events
|> Enum.each(fn
  {:feed, feed} -> save_feed_metadata(feed)
  {:item, item} -> process_item(item)
  {:error, reason} -> handle_parse_error(reason)
end)

The local file remains caller-owned, even if enumeration stops early. The parser runs on demand and only advances as events are requested. :chunk_size controls input chunks and SAX character-data callback sizing.

Stream a remote URL

{:ok, session} = FeedReader.stream("https://example.com/feed.xml")

try do
  session.events
  |> Enum.take(20)
  |> Enum.each(&process_event/1)
after
  FeedReader.close(session)
end

Enum.take/2 limits events, including the initial feed-metadata event; it does not count only item events.

Remote sessions include a fetch record. The remote response is downloaded, decoded, checksummed, and staged in an owned temporary file before stream/2 returns. Parsing the staged file is lazy; the network download is not. Fully consuming events removes the temporary file. If enumeration stops early, raises, or otherwise may not finish, call FeedReader.close/1 in an after block. Closing is safe after completion and can be repeated.

An ETag 304 or a matching previous checksum returns {:ok, :not_modified} or {:ok, :not_changed} respectively, instead of a session.

Collect input chunks into a result

parse_stream/2 is different from stream_file/2: it takes XML chunks but collects the result into memory, optionally stopping parsing at :max_items. Chunks must be binaries. The chunks are staged to a temporary file before parsing, so use stream_file/2 or stream/2 to process large feeds incrementally.

chunks = File.stream!("path/to/feed.xml", 16_384, [:raw])
{:ok, result} = FeedReader.parse_stream(chunks)

Chunk input is staged in a library-owned temporary file and cleaned up on success and failure. If a supplied item cap is reached, the result's item_limit_reached? field is true, and later parser errors are not observed. Use stream_file/2 when you want to process events without collecting all items.

Validate before exposing events

stream_file_verified/2 and stream_verified/2 validate the feed before returning an event session. This is useful when consumers must not see any events unless parsing succeeds. The validation pass adds work; delivery still parses the source to emit events. :max_parse_passes bounds validation and delivery passes and defaults to 5 (a value below 2 cannot allow both passes).

{:ok, session} = FeedReader.stream_file_verified("path/to/feed.xml")
Enum.each(session.events, &process_event/1)

Verified remote sessions have the same temporary-file ownership rules as stream/2: close them in after when enumeration may end early. Like the other parser APIs, parsing stops when the feed root closes; trailing content after a complete root is not parsed as part of the feed.