# RustyXML Architecture
A purpose-built Rust NIF for ultra-fast XML parsing in Elixir. Not a wrapper around an existing library—custom-built from the ground up for optimal BEAM integration with full XPath 1.0 support. Drop-in replacement for both SweetXml and Saxy.
## Key Innovations
### Purpose-Built, Not Wrapped
Unlike projects that wrap existing Rust crates (like quick-xml or roxmltree), RustyXML is **designed specifically for Elixir**:
- **Direct BEAM term construction** — Results go straight to Erlang terms, no intermediate serialization
- **ResourceArc integration** — Documents and streaming parser state managed by BEAM's garbage collector
- **Dirty scheduler awareness** — All raw-XML parse NIFs run on dirty CPU schedulers
- **Zero-copy where possible** — Span-based references into original input, only allocates for entity decoding
- **Structural index** — Cache-friendly storage with compact span structs and flat arrays
### Unified Architecture
RustyXML v0.2.0 consolidated multiple parsing strategies into a single optimized path: the **structural index**. A single `UnifiedScanner` tokenizes input once, dispatching to a `ScanHandler` trait that builds the appropriate representation:
| Path | Description | Best For |
|------|-------------|----------|
| `parse/1` + `xpath/2` | Structural index with XPath | General XML processing |
| `stream_tags/3` | Bounded-memory streaming | Large files (GB+) |
| `sax_parse/1` | SAX event collection | Event-driven processing |
All three paths share the same SIMD-accelerated scanner and well-formedness validation.
### Memory Efficiency
- **Structural index** — Elements stored as compact span structs (32 bytes each) referencing the original input
- **Zero-copy strings** — Tag names, attribute values, and text stored as `(offset, length)` spans
- **Sub-binary returns** — BEAM sub-binaries share memory with the original input
- **Streaming bounded memory** — Process 10GB+ files with ~128 KB combined NIF + BEAM peak via zero-copy tokenization and direct BEAM binary encoding
- **mimalloc allocator** — High-performance allocator for reduced fragmentation
- **Optional memory tracking** — Opt-in profiling with zero overhead when disabled
### Validated Correctness
- **100% W3C/OASIS XML Conformance** — All 1089 applicable tests pass (218 valid + 871 not-well-formed rejections), verified individually against the official [xmlconf](https://www.w3.org/XML/Test/) suite
- **1296+ tests** including the full conformance suite, batch accessor clamping, and lazy XPath coverage
- **Cross-path validation** — All paths produce consistent output
- **SweetXml compatibility** — Verified identical behavior for common API patterns
---
## Quick Start
```elixir
import RustyXML
xml = """
Elixir in Action45.00
Programming Phoenix50.00
"""
# Get all books
RustyXML.xpath(xml, ~x"//book"l)
# Get text content
RustyXML.xpath(xml, ~x"//title/text()"s)
# Extract multiple values
RustyXML.xmap(xml, [
titles: ~x"//title/text()"sl,
prices: ~x"//price/text()"sl
])
```
---
## Core Architecture
### UnifiedScanner and ScanHandler
The `UnifiedScanner` is the single entry point for all XML tokenization. It uses `memchr`-based SIMD scanning to find delimiters, then dispatches events through the `ScanHandler` trait:
```
XML Input
|
v
UnifiedScanner (memchr SIMD tokenization)
|
+---> IndexBuilder (ScanHandler) ---> StructuralIndex ---> XPath
|
+---> SaxCollector (ScanHandler) ---> SAX Events
|
+---> StreamingParser ---> Complete Elements
```
The `ScanHandler` trait:
```rust
trait ScanHandler {
fn start_element(&mut self, name: Span, attrs: &[(Span, Span)], is_empty: bool);
fn end_element(&mut self, name: Span);
fn text(&mut self, span: Span, needs_entity_decode: bool);
fn cdata(&mut self, span: Span);
fn comment(&mut self, span: Span);
fn processing_instruction(&mut self, target: Span, data: Option);
}
```
Adding a new processing mode requires only implementing the trait—no changes to the scanner.
### Structural Index
The structural index is the core document representation. Instead of building a DOM tree with string copies, it stores compact structs that reference byte offsets into the original input:
```rust
struct Span {
offset: u32,
len: u16, // 6 bytes total
}
struct IndexElement { // 32 bytes
name: Span,
ns_prefix: Option,
parent: u32,
children: Range, // into flat children_data array
attrs: Range, // into flat attrs array
}
struct IndexText { // 16 bytes
span: Span,
parent: u32,
needs_entity_decode: bool,
}
struct IndexAttribute { // 12 bytes
name: Span,
value: Span,
}
```
**Memory profile for 2.93 MB document:**
- Structural index: **12.8 MB** (4.4x input size)
- Old DOM approach: **30.2 MB** (10.3x input size)
- SweetXml/xmerl: allocated entirely on BEAM heap
The `IndexedDocumentView` implements the `DocumentAccess` trait, allowing the XPath engine to evaluate queries on the structural index without any conversion step.
### SIMD-Accelerated Scanning
Tag and content boundary detection uses `memchr` for hardware-accelerated scanning:
```rust
use memchr::{memchr, memchr2, memchr3};
// Find next tag start — SIMD accelerated
fn find_tag_start(input: &[u8], pos: usize) -> Option {
memchr(b'<', &input[pos..]).map(|i| pos + i)
}
// Content scanning for entities and markup
fn find_content_break(input: &[u8], pos: usize) -> Option {
memchr3(b'<', b'&', b']', &input[pos..])
}
```
**SIMD support:** SSE2 (x86_64 default), AVX2 (runtime detect), NEON (aarch64), simd128 (wasm)
---
## Parsing
### Standard Parse (`parse/1`)
All parsing flows through the structural index:
```elixir
doc = RustyXML.parse("