Parquex.Stress (parquex v0.4.0)

View Source

Repeatedly scans a realistic Parquet dataset and reports memory behavior.

Each complete pass runs in a short-lived process. Process exit releases the store, streams, batches, and native reader resources before the next pass. The report separates BEAM-managed memory from operating-system RSS and records native active-resource counts after every iteration.

The default path is tmp/stress_reference/event_log. That directory is gitignored and is intended for a local copy of realistic data.

Parquex.Stress.run(
  iterations: 100,
  warmup: 3,
  batch_size: 4_096
)

Pass path: to scan one .parquet file or every Parquet file beneath a directory in deterministic key order. run/1 prints compact progress and returns only :ok. Use measure/1 when you need the detailed report as a map.

Summary

Types

iteration_sample()
memory_sample()

Functions

measure(options \\ [])

Runs repeated scans and returns the complete measurement report.

measure(path, options)

Runs repeated scans beneath path and returns the complete measurement report.

run(options \\ [])

Runs repeated scans with dot progress, discards the report, and returns :ok.

run(path, options)

Runs repeated scans beneath path and returns only :ok.

sample_memory()

Forces collection in the caller and returns a fresh memory sample.

Types

iteration_sample()

@type iteration_sample() :: %{
  iteration: pos_integer(),
  rows: non_neg_integer(),
  batches: non_neg_integer(),
  range_bytes: non_neg_integer(),
  peak_buffered_bytes: non_neg_integer(),
  elapsed_ms: non_neg_integer(),
  memory: memory_sample(),
  resources: map()
}

memory_sample()

@type memory_sample() :: %{
  beam: non_neg_integer(),
  processes: non_neg_integer(),
  binary: non_neg_integer(),
  ets: non_neg_integer(),
  rss: non_neg_integer() | nil
}

Functions

measure(options \\ [])

@spec measure(keyword()) :: map()

Runs repeated scans and returns the complete measurement report.

measure(path, options)

@spec measure(
  Path.t(),
  keyword()
) :: map()

Runs repeated scans beneath path and returns the complete measurement report.

run(options \\ [])

@spec run(keyword()) :: :ok

Runs repeated scans with dot progress, discards the report, and returns :ok.

run(path, options)

@spec run(
  Path.t(),
  keyword()
) :: :ok

Runs repeated scans beneath path and returns only :ok.

sample_memory()

@spec sample_memory() :: memory_sample()

Forces collection in the caller and returns a fresh memory sample.