PropertyDamage.Model.Projection.Statistics (PropertyDamage v0.2.0)

View Source

Projection that computes statistical properties over event streams.

Traditional assertions check exact conditions. Statistical projections enable probabilistic assertions like:

  • "p99 latency < 100ms"
  • "error rate < 1%"
  • "success rate > 99% over last 100 operations"

Why Statistical Projections?

Nemesis can inject latency, but we need statistical assertions to verify the system handles it gracefully:

  • A single slow request isn't a bug
  • 50% of requests being slow IS a bug
  • 1% error rate might be acceptable; 10% is not

Usage

defmodule MyModel do
  def assertion_projections do
    [
      {PropertyDamage.Model.Projection.Statistics, [
        window_size: 100,
        assertions: [
          {:p99_latency_ms, :less_than, 500},
          {:error_rate, :less_than, 0.05},
          {:success_rate, :greater_than, 0.95}
        ]
      ]}
    ]
  end
end

Tracked Metrics

  • :p50_latency_ms - Median latency
  • :p95_latency_ms - 95th percentile latency
  • :p99_latency_ms - 99th percentile latency
  • :max_latency_ms - Maximum observed latency
  • :mean_latency_ms - Average latency
  • :success_rate - Ratio of successes to total
  • :error_rate - Ratio of errors to total
  • :throughput - Operations per second (requires time tracking)

Recording Metrics

Events should include latency information. The projection looks for:

  • :latency_ms or :duration_ms fields for latency tracking
  • Success events vs error events for rate calculation

You can also record metrics explicitly in your adapter.

Limitations

  • Statistics only meaningful with sufficient sample size
  • Shrinking statistical failures is problematic (minimal case may not reproduce)
  • Thresholds are environment-dependent (CI vs production hardware)

Summary

Functions

Check all configured assertions against current metrics.

Compute all metrics from current state.

Format statistics as a human-readable string.

Record an error/failure.

Record a latency sample.

Record a successful operation.

Get a summary of current statistics.

Types

assertion()

@type assertion() :: {metric(), comparator(), number()}

comparator()

@type comparator() :: :less_than | :greater_than | :equal_to

metric()

@type metric() ::
  :p50_latency_ms
  | :p95_latency_ms
  | :p99_latency_ms
  | :max_latency_ms
  | :mean_latency_ms
  | :success_rate
  | :error_rate
  | :total_count

t()

@type t() :: %PropertyDamage.Model.Projection.Statistics{
  assertions: [assertion()],
  current_step: non_neg_integer(),
  error_count: non_neg_integer(),
  latency_samples: :queue.queue(float()),
  start_time: integer(),
  success_count: non_neg_integer(),
  window_size: pos_integer()
}

Functions

check_assertions(state)

@spec check_assertions(t()) :: :ok | {:error, String.t()}

Check all configured assertions against current metrics.

compute_metrics(state)

@spec compute_metrics(t()) :: %{required(metric()) => number()}

Compute all metrics from current state.

format_summary(state)

@spec format_summary(t()) :: String.t()

Format statistics as a human-readable string.

record_error(state)

@spec record_error(t()) :: t()

Record an error/failure.

record_latency(state, latency_ms)

@spec record_latency(t(), number()) :: t()

Record a latency sample.

record_success(state)

@spec record_success(t()) :: t()

Record a successful operation.

summary(state)

@spec summary(t()) :: map()

Get a summary of current statistics.