Collection Preparation and Import

Copy Markdown View Source

Benchmark

Collection preparation and complete filesystem import over increasing synthetic corpora. Setup and publication are outside timed work. Import includes bounded reads, JSON decoding, canonical hashes, shape validation and indexed provenance reconstruction. Memory is BEAM allocation measured by Benchee, not peak RSS. These are local measurements, not performance thresholds.

System

Benchmark suite executing on the following system:

Operating System macOS
CPU Information Apple M5 Pro
Number of Available Cores 18
Available Memory 48 GB
Elixir Version 1.18.4
Erlang Version 27.3.4.15

Configuration

Benchmark suite executing with the following configuration:

:time 3 s
:parallel 1
:warmup 1 s

Statistics

Input: 1000 documents

Run Time

Name IPS Average Deviation Median 99th %
Collection.prepare/2 394.19 2.54 ms ±6.27% 2.48 ms 2.96 ms
Collection.load/1 50.30 19.88 ms ±1.86% 19.84 ms 21.64 ms

Run Time Comparison

Name IPS Slower
Collection.prepare/2 394.19  
Collection.load/1 50.30 7.84x

Memory Usage

Name Average Factor
Collection.prepare/2 6.75 MB  
Collection.load/1 41.61 MB 6.17x

Input: 16000 documents

Run Time

Name IPS Average Deviation Median 99th %
Collection.prepare/2 16.32 61.28 ms ±1.20% 61.21 ms 64.83 ms
Collection.load/1 1.43 697.78 ms ±7.72% 687.29 ms 778.24 ms

Run Time Comparison

Name IPS Slower
Collection.prepare/2 16.32  
Collection.load/1 1.43 11.39x

Memory Usage

Name Average Factor
Collection.prepare/2 109.82 MB  
Collection.load/1 672.02 MB 6.12x

Input: 4000 documents

Run Time

Name IPS Average Deviation Median 99th %
Collection.prepare/2 96.21 10.39 ms ±1.75% 10.38 ms 11.01 ms
Collection.load/1 7.69 130.01 ms ±2.41% 131.33 ms 135.35 ms

Run Time Comparison

Name IPS Slower
Collection.prepare/2 96.21  
Collection.load/1 7.69 12.51x

Memory Usage

Name Average Factor
Collection.prepare/2 27.13 MB  
Collection.load/1 167.11 MB 6.16x

Input: 8000 documents

Run Time

Name IPS Average Deviation Median 99th %
Collection.prepare/2 43.56 22.96 ms ±2.36% 22.97 ms 25.47 ms
Collection.load/1 3.71 269.22 ms ±1.01% 268.85 ms 276.30 ms

Run Time Comparison

Name IPS Slower
Collection.prepare/2 43.56  
Collection.load/1 3.71 11.73x

Memory Usage

Name Average Factor
Collection.prepare/2 54.64 MB  
Collection.load/1 335.02 MB 6.13x