Testing and benchmarks

Test layers

The suite deliberately separates failures by ownership:

The default HTTP suite includes the real in-process gateway and excludes only external SDK suites:

mix test apps/ferricstore_http/test
FERRICSTORE_PYTHON_SDK_PATH=../ferricstore-python \
  mix test apps/ferricstore_http/test/ferricstore_http/python_sdk_integration_test.exs \
  --include python_sdk_integration

The Python executable must have the SDK development extras installed. Set FERRICSTORE_PYTHON_EXECUTABLE when that executable is not python3.

In-memory HTTP/1.1 benchmark

This benchmark isolates listener, parsing, authentication-boundary, encoding, and keep-alive costs from FerricStore storage work:

mix run --no-start bench/http/keepalive_benchmark.exs \
  --clients 50 \
  --requests-per-client 100 \
  --commands-per-request 1 \
  --warmup 5

Each worker represents one warm function environment with one reused HTTP/1.1 connection. Large simultaneous connection bursts also depend on the host listen-backlog and file-descriptor limits. max_connections and acceptors size the listener but cannot override kernel caps; tune the host and distribute bursts across proxy instances when validating thousands of environments.

Python SDK Lambda benchmark

This benchmark starts a TLS listener backed by the real in-process FerricStore gateways, creates one ACL identity per function group, and runs concurrent function environments. Warm scenarios create one SDK client per environment and reuse it. Cold scenarios create a client and TLS connection for every request.

export FERRICSTORE_PYTHON_SDK_PATH=../ferricstore-python
export FERRICSTORE_PYTHON_EXECUTABLE=/path/to/python-with-sdk-dev-extras
export FERRICSTORE_LAMBDA_GROUPS=8
export FERRICSTORE_LAMBDA_ENVS_PER_GROUP=8
export FERRICSTORE_LAMBDA_WARM_REQUESTS=250
export FERRICSTORE_LAMBDA_COLD_REQUESTS=4
export FERRICSTORE_LAMBDA_BENCH_WORKLOAD=flow_get
export FERRICSTORE_LAMBDA_BENCH_TRANSPORTS=http,http2

mix run --no-start bench/http/python_sdk_lambda_benchmark.exs

Supported controls are:

Every result is emitted as one JSON object containing the workload, concurrency, request, command, and error counts, duration, requests and commands per second, and p50/p95/p99/max request latency. The batch and flow_batch workloads count one HTTP request and two FerricStore commands per operation. The Flow workloads use the real FerricStore gateway; they measure storage and workflow execution as well as the HTTP transport. flow_start specifically covers the SDK's structured native-opcode bridge. Diagnostic transport implementations (http_transport, httpclient, and httpx) intentionally support only PING; use http, http2, or native for data workloads. Any request error makes the benchmark exit unsuccessfully. Compare runs on the same host and toolchain; these numbers are regression and capacity-planning evidence, not a universal production limit. The isolated benchmark raises the shared authentication limiter only to the configured concurrency plus headroom, preventing a burst of successful simultaneous authentications from obscuring transport measurements. The normal HTTP integration suite still exercises the production rate-limit response.

Local optimization evidence

On the same 16-scheduler development host, with 64 process-isolated warm environments, TLS, Basic authentication, the real FerricStore gateway, and 250 timed PING requests per environment:

The two matching post-change runs are about 59% faster than the sustained pre-change run. Later samples on this workstation were rejected because unrelated long-running benchmark processes and a virtual machine consumed multiple cores. Always inspect host contention, repeat the same scenario, and retain controls; do not select the best sample from a noisy machine. The release-candidate rows validate the complete TLS and Python SDK path on the newer toolchain; they are not a causal comparison with the older scoped-lease measurements.

Homogeneous command-gateway batch evidence

The Java SDK's real HTTP KV benchmark also covers the shared gateway and durable storage rather than an in-memory backend. The following controlled A/B used the unchanged 0.11.11 worktree and the candidate worktree on the same host, plain HTTP/1.1, 64 in-flight batches, 100 hot keys, and 16-byte values. GET batches contained 1,000 commands and SET batches contained 500 commands. Each value is the median of three zero-error five-second samples after the binary correctness probe. SET candidate samples used a fresh server process so sustained writes from an earlier sample could not trigger storage-batcher admission in a later sample. The benchmark-only authentication attempt allowance was 1,000 on both A/B servers, isolating the gateway and storage path from connection-startup auth bursts.

The same candidate server then completed the 10,000-flow, three-step Java workflow benchmark with zero execution or verification errors: 2,576 workflow completions/s on Java 21 and 2,608/s on Java

17. The workflow commands deliberately remain on their existing structured paths, so these runs are a regression guard rather than evidence that the KV fast path accelerates Flow. These are local regression measurements, not universal capacity claims; compare candidates on the same host and retain the zero-error requirement.

With the production default of 10 authentication attempts per window, a fresh Java HTTP/1.1 client opening 64 connections concurrently also intermittently received 429 rate_limited for valid Basic credentials. The retained throughput samples exclude that separate startup-admission effect; it needs its own auth single-flight or queued-verification correction rather than a higher production rate limit.