defmodule PropertyDamage do @moduledoc """ PropertyDamage: A stateful property-based testing framework for Elixir. PropertyDamage combines the power of property-based testing with stateful system testing, allowing you to verify that your system behaves correctly under any sequence of operations. ## Overview Traditional property-based testing generates random inputs and verifies properties hold for all inputs. Stateful property-based testing extends this by generating random *sequences of operations* (commands) and verifying that the system under test (SUT) behaves correctly throughout the entire sequence. ## Key Concepts - **Commands**: Operations that can be executed against the SUT (create, update, delete, etc.) - **Model**: Defines what commands are available and how state is tracked - **Projections**: Pure state reducers that process commands and events to maintain state - **Adapters**: Bridge between the test framework and the actual SUT - **Refs**: Symbolic placeholders for entity IDs, resolved during execution ## Two-Phase Execution PropertyDamage uses a two-phase execution model: 1. **Symbolic Phase**: Generate a sequence of commands with symbolic refs 2. **Concrete Phase**: Execute commands against the SUT, resolving refs to real values This separation enables powerful shrinking of failing test cases while maintaining the dependency relationships between commands. ## Basic Usage defmodule MyModelTest do use ExUnit.Case use PropertyDamage @model MyApp.TestModel @adapter MyApp.TestAdapter property_damage "system maintains invariants" do max_commands: 50, max_runs: 100 end end ## Running Directly PropertyDamage.run( model: MyApp.TestModel, adapter: MyApp.TestAdapter, max_commands: 50, max_runs: 100 ) ## Debugging Failures When a test fails, PropertyDamage provides rich tools for understanding what went wrong: {:error, failure} = PropertyDamage.run(model: M, adapter: A) # Understand why each command in the shrunk sequence is needed explanation = PropertyDamage.explain(failure) # Find the specific field/value that caused the failure {:ok, trigger} = PropertyDamage.isolate_trigger(failure) # Generate a reproducible test case test_code = PropertyDamage.generate_test(failure, format: :exunit) # Try harder to shrink if needed {:ok, smaller} = PropertyDamage.shrink_further(failure, strategy: :exhaustive) # Replay step-by-step {:ok, steps} = PropertyDamage.replay(failure) ## Failure Persistence Save failures for later analysis or regression testing: {:ok, path} = PropertyDamage.save_failure(failure, "failures/") {:ok, loaded} = PropertyDamage.load_failure(path) failures = PropertyDamage.list_failures("failures/") See `PropertyDamage.Persistence` for details. ## Seed Library Track interesting seeds for regression testing: {:ok, library} = PropertyDamage.load_seed_library("seeds.json") {:ok, library} = PropertyDamage.add_to_seed_library(library, failure, tags: [:bug]) PropertyDamage.save_seed_library(library, "seeds.json") See `PropertyDamage.SeedLibrary` for details. ## Coverage Metrics Track how thoroughly your model is being exercised: coverage = PropertyDamage.coverage(result, MyModel) IO.puts(PropertyDamage.Coverage.format(coverage)) See `PropertyDamage.Coverage` for details. ## Flakiness Detection Detect non-deterministic behavior in your SUT: PropertyDamage.check_determinism(Model, Adapter, seed, runs: 10) flaky = PropertyDamage.discover_flaky_seeds(Model, Adapter, num_seeds: 20) See `PropertyDamage.Flakiness` for details. ## Architecture The framework consists of several layers: - **Tier 0 (Core Types)**: Ref, Command, Projection, Model behaviours - **Tier 1 (Execution)**: Adapter, EventQueue, InjectorAdapter, Executor - **Tier 2 (Shrinking)**: Validator, Shrinker, dependency graph - **Tier 3 (Analysis)**: Analysis, Replay, Coverage, Flakiness - **Utilities**: Persistence, SeedLibrary, mix tasks See the individual module documentation for detailed information on each component. """ alias PropertyDamage.{ Generator, Executor, Shrinker, Validation, Options, EventQueue, Sequence, Stutter, FailureReport, Progress, Telemetry } alias PropertyDamage.Shrinker.Config, as: ShrinkerConfig @typedoc """ Result statistics from a successful run. """ @type stats :: %{ runs: non_neg_integer(), total_commands: non_neg_integer(), seed: integer() } @typedoc """ Failure report from a failed run. """ @type failure_report :: %{ seed: integer(), run_number: non_neg_integer(), original_sequence: Sequence.t(), shrunk_sequence: Sequence.t(), failed_at_index: non_neg_integer(), failure_reason: term(), shrink_iterations: non_neg_integer(), shrink_time_ms: non_neg_integer() } @typedoc """ Result from `run/1` - either success stats or a failure report. """ @type result :: {:ok, stats()} | {:error, failure_report()} @doc """ Run a property-based test. This is the main entry point for PropertyDamage. It generates command sequences, executes them against the SUT, and shrinks failures to minimal reproductions. ## Required Options - `:model` - Model module implementing PropertyDamage.Model - `:adapter` - Adapter module implementing PropertyDamage.Adapter ## Optional Options - `:max_commands` - Maximum commands per sequence (default: 50) - `:max_runs` - Number of test sequences to run (default: 100) - `:seed` - Random seed for reproducibility (default: random) - `:injector_adapters` - List of InjectorAdapter modules (default: []) - `:adapter_config` - Config passed to adapter.setup/1 (default: %{}) - `:shrink` - Whether to shrink failing sequences (default: true) - `:shrinker_config` - ShrinkerConfig struct for tuning shrinking - `:on_failure` - Callback function receiving failure_report (default: nil) - `:regression` - Keyword list for automatic regression test management (see below) - `:verbose` - Print progress and configuration (default: false) - `:validate` - Run configuration validation first (default: true) - `:branching` - Keyword list for parallel branching (see below) - `:stutter` - Map for idempotency testing (see below) ## Branching Options Pass `branching: [...]` to generate branching (parallel) sequences: - `:branch_probability` - Probability of creating a branch point (default: 0.2) - `:max_branches` - Maximum number of parallel branches (default: 3) - `:max_branch_length` - Maximum commands per branch (default: 5) - `:min_prefix_length` - Minimum commands before branching (default: 3) Branching sequences enable detection of race conditions by executing commands in parallel branches and checking linearizability. ## Stutter Options (Idempotency Testing) Pass `stutter: %{...}` to enable idempotency testing: - `:probability` - Probability of stuttering each command (default: 0.1) - `:max_repeats` - Maximum retry attempts per stuttered command (default: 2) - `:delay_ms` - Delay between retries, `{min, max}` tuple or integer (default: {0, 100}) - `:commands` - `:all` or list of command modules to stutter (default: :all) - `:comparison` - Event comparison mode (default: :strict) - `:strict` - Events must be exactly equal - `{:structural, fields}` - Ignore specified fields when comparing - `{:custom, fun}` - Custom comparison function `fn(events1, events2) -> :match | {:mismatch, map()}` Stutter testing verifies that retrying commands produces consistent results (idempotency). Retry events are captured but not applied to projections. ## Regression Options Pass `regression: [...]` to automatically save failures for regression testing: - `:save_failures` - Directory to save failure files - `:seed_library` - Path to seed library JSON file - `:generate_tests` - Directory to generate ExUnit test files - `:tags` - Tags to add to seed library entries (default: `[:auto_detected]`) - `:dedup` - Skip if similar failure exists (default: false) - `:dedup_threshold` - Similarity threshold for dedup (default: 0.90) - `:verbose` - Print regression actions (default: false) This option integrates with `:on_failure` - both can be used together. ## Returns - `{:ok, stats}` - All runs passed - `{:error, failure_report}` - A run failed ## Examples # Basic usage PropertyDamage.run(model: MyModel, adapter: MyAdapter) # With options PropertyDamage.run( model: MyModel, adapter: MyAdapter, max_commands: 100, max_runs: 1000, seed: 12345 ) # With failure callback PropertyDamage.run( model: MyModel, adapter: MyAdapter, on_failure: fn failure_report -> IO.puts("Failed at command \#{failure_report.failed_at_index}") end ) # With automatic regression management PropertyDamage.run( model: MyModel, adapter: MyAdapter, regression: [ save_failures: "failures/", seed_library: "seeds.json", generate_tests: "test/regressions/", dedup: true ] ) """ @spec run(keyword()) :: {:ok, stats()} | {:error, failure_report()} def run(opts) do # Validate options with NimbleOptions - applies defaults and provides helpful errors opts = Options.validate_run!(opts) model = opts[:model] adapter = opts[:adapter] max_commands = opts[:max_commands] max_runs = opts[:max_runs] seed = opts[:seed] || :rand.uniform(1_000_000_000) injector_adapters = opts[:injector_adapters] adapter_config = opts[:adapter_config] shrink = opts[:shrink] shrinker_config = opts[:shrinker_config] || ShrinkerConfig.new() on_failure = build_on_failure_callback(opts) verbose = opts[:verbose] validate = opts[:validate] branching = opts[:branching] stutter_config = Stutter.parse_config(opts[:stutter]) # Validate configuration if validate do {:ok, warnings} = Validation.validate!(model, adapter, injector_adapters: injector_adapters) if verbose do Validation.print_summary(model, adapter, warnings) end end # Print runtime warnings when verbose if verbose do runtime_warnings = Validation.runtime_warnings(opts) unless Enum.empty?(runtime_warnings) do IO.puts("Runtime Warnings:") for warning <- runtime_warnings do IO.puts(" ⚠ #{warning}") end IO.puts("") end # Print test run header Progress.print_header(model, adapter, opts) end # Setup once (if model implements it) setup_once_result = if function_exported?(model, :setup_once, 1) do model.setup_once(%{adapter_config: adapter_config}) else :ok end case setup_once_result do :ok -> # Emit telemetry for run start telemetry_metadata = %{ model: model, adapter: adapter, max_runs: max_runs, max_commands: max_commands, seed: seed } start_time = System.system_time() Telemetry.run_start(telemetry_metadata) try do result = do_run( model, adapter, max_commands, max_runs, seed, injector_adapters, adapter_config, shrink, shrinker_config, on_failure, verbose, branching, stutter_config ) # Emit telemetry for run stop {result_type, result_data} = case result do {:ok, stats} -> {:ok, stats} {:error, _} -> {:error, %{}} end Telemetry.run_stop( start_time, Map.merge(telemetry_metadata, %{ result: result_type, runs_completed: if(result_type == :ok, do: result_data[:runs], else: 0), total_commands: if(result_type == :ok, do: result_data[:total_commands], else: 0) }) ) result rescue e -> Telemetry.run_exception(start_time, :error, e, __STACKTRACE__, telemetry_metadata) reraise e, __STACKTRACE__ after # Teardown once if function_exported?(model, :teardown_once, 1) do model.teardown_once(%{}) end end {:error, reason} -> {:error, %{setup_once_failed: reason}} end end defp do_run( model, adapter, max_commands, max_runs, seed, injector_adapters, adapter_config, shrink, shrinker_config, on_failure, verbose, branching, stutter_config ) do # Seed the RNG :rand.seed(:exsss, {seed, seed, seed}) # Generate sequences and run generator_opts = [max_commands: max_commands] generator_opts = if branching, do: Keyword.put(generator_opts, :branching, branching), else: generator_opts generator = Generator.generate_sequence(model, generator_opts) run_loop( generator, model, adapter, max_runs, seed, injector_adapters, adapter_config, shrink, shrinker_config, on_failure, verbose, stutter_config, 0, 0 ) end defp run_loop( _generator, _model, _adapter, max_runs, seed, _injector_adapters, _adapter_config, _shrink, _shrinker_config, _on_failure, verbose, _stutter_config, run_number, total_commands ) when run_number >= max_runs do stats = %{runs: max_runs, total_commands: total_commands, seed: seed} if verbose do Progress.print_success(stats) end {:ok, stats} end defp run_loop( generator, model, adapter, max_runs, seed, injector_adapters, adapter_config, shrink, shrinker_config, on_failure, verbose, stutter_config, run_number, total_commands ) do # Generate a command sequence sequence = generate_one(generator) command_count = Sequence.command_count(sequence) if verbose do Progress.print_run(run_number, max_runs, sequence) end # Emit telemetry for sequence start seq_start_time = System.system_time() Telemetry.sequence_start(%{ run_number: run_number, command_count: command_count, branching: Sequence.branching?(sequence) }) # Setup each (if model implements it) setup_each_result = if function_exported?(model, :setup_each, 1) do model.setup_each(%{adapter_config: adapter_config, run_number: run_number}) else :ok end case setup_each_result do :ok -> # Start event queue for injectors {:ok, event_queue} = EventQueue.start_link() # Setup injector adapters setup_injectors(injector_adapters, event_queue) try do # Execute the sequence {:ok, result} = Executor.run(sequence, model, adapter, adapter_config: adapter_config, event_queue: event_queue, stutter_config: stutter_config ) # Emit telemetry for sequence stop Telemetry.sequence_stop(seq_start_time, %{ run_number: run_number, success: result.success, commands_executed: command_count }) if result.success do # Success - continue to next run run_loop( generator, model, adapter, max_runs, seed, injector_adapters, adapter_config, shrink, shrinker_config, on_failure, verbose, stutter_config, run_number + 1, total_commands + command_count ) else # Failure - shrink and report handle_failure( sequence, result, model, adapter, adapter_config, event_queue, shrink, shrinker_config, on_failure, verbose, seed, run_number ) end after # Teardown injectors teardown_injectors(injector_adapters) EventQueue.stop(event_queue) # Teardown each if function_exported?(model, :teardown_each, 1) do model.teardown_each(%{}) end end {:error, reason} -> {:error, %{setup_each_failed: reason, run_number: run_number}} end end defp generate_one(generator) do # Use StreamData's internal generation to get a single value case Enumerable.reduce(generator, {:cont, nil}, fn val, _ -> {:halt, val} end) do {:halted, value} -> value {:done, _} -> [] end end defp setup_injectors(injector_adapters, event_queue) do for adapter <- injector_adapters do if function_exported?(adapter, :setup, 1) do adapter.setup(%{event_queue: event_queue}) end end end defp teardown_injectors(injector_adapters) do for adapter <- injector_adapters do if function_exported?(adapter, :teardown, 1) do adapter.teardown(%{}) end end end defp handle_failure( sequence, result, model, adapter, adapter_config, event_queue, shrink, shrinker_config, on_failure, verbose, seed, run_number ) do # Skip shrinking for stutter-related failures since: # 1. The failure is about SUT idempotency, not the command sequence # 2. Shrinking without stutter won't reproduce the failure should_shrink = shrink and not stutter_failure?(result.failure_reason) {shrunk_sequence, shrink_iterations, shrink_time_ms} = if should_shrink do shrink_result = Shrinker.shrink(sequence, failed_at_index: result.failed_at_index, failure_reason: result.failure_reason, model: model, adapter: adapter, adapter_config: adapter_config, config: shrinker_config, event_queue: event_queue ) {shrink_result.sequence, shrink_result.iterations, shrink_result.time_ms} else {sequence, 0, 0} end # Re-execute shrunk sequence to get fresh event log and state # (the original result has state from before shrinking) {:ok, fresh_result} = Executor.run(shrunk_sequence, model, adapter, adapter_config: adapter_config, event_queue: event_queue ) # Create rich failure report with fresh state from shrunk sequence failure_report = FailureReport.new( seed: seed, run_number: run_number, original_sequence: sequence, shrunk_sequence: shrunk_sequence, failed_at_index: fresh_result.failed_at_index, failure_reason: fresh_result.failure_reason, shrink_iterations: shrink_iterations, shrink_time_ms: shrink_time_ms, event_log: fresh_result.event_log, projections: fresh_result.projections, projections_before: fresh_result.projections_before, refs: fresh_result.refs, model: model, adapter: adapter, linearization: fresh_result.linearization ) # Print failure summary when verbose if verbose do Progress.print_failure(failure_report) end if on_failure do on_failure.(failure_report) end {:error, failure_report} end # Check if a failure reason is stutter-related (idempotency violation or execution failure) defp stutter_failure?({:idempotency_violation, _}), do: true defp stutter_failure?({:stutter_execution_failed, _}), do: true defp stutter_failure?(_), do: false # Build the on_failure callback from :on_failure and :regression options defp build_on_failure_callback(opts) do on_failure = Keyword.get(opts, :on_failure) regression = Keyword.get(opts, :regression) cond do # Both options specified - compose them on_failure != nil and regression != nil -> regression_handler = PropertyDamage.Regression.handler(regression) fn failure_report -> on_failure.(failure_report) regression_handler.(failure_report) end # Only on_failure specified on_failure != nil -> on_failure # Only regression specified regression != nil -> PropertyDamage.Regression.handler(regression) # Neither specified true -> nil end end @doc """ Attempt further shrinking on an existing failure report. Use this when the initial shrinking didn't produce a minimal enough sequence. You can specify more aggressive time/iteration limits or different strategies. ## Options - `:max_iterations` - Maximum shrink attempts (default: 5000) - `:max_time_ms` - Maximum time for shrinking in ms (default: 60000) - `:strategy` - Shrinking strategy (default: :thorough) - `:quick` - Fast shrinking, may miss some reductions - `:thorough` - Balanced approach (default) - `:exhaustive` - Try all possible reductions (slow) - `:shrink_arguments` - Whether to shrink argument values (default: true) - `:adapter_config` - Adapter configuration (uses report's adapter if not specified) ## Returns - `{:ok, new_failure_report}` - Shrinking succeeded, possibly smaller sequence - `{:error, reason}` - Shrinking failed (e.g., missing model/adapter) ## Example {:error, failure} = PropertyDamage.run(model: M, adapter: A) # Try harder to shrink {:ok, smaller} = PropertyDamage.shrink_further(failure, max_time_ms: 120_000, strategy: :exhaustive ) IO.puts("Reduced from \#{length(original)} to \#{length(smaller)} commands") """ @spec shrink_further(FailureReport.t(), keyword()) :: {:ok, FailureReport.t()} | {:error, term()} def shrink_further(%FailureReport{} = report, opts \\ []) do model = report.model adapter = report.adapter if is_nil(model) or is_nil(adapter) do {:error, :missing_model_or_adapter} else adapter_config = Keyword.get(opts, :adapter_config, %{}) # Build shrinker config from options strategy = Keyword.get(opts, :strategy, :thorough) shrinker_config = ShrinkerConfig.new( max_iterations: strategy_iterations(strategy, opts), max_time_ms: strategy_time(strategy, opts), shrink_arguments: Keyword.get(opts, :shrink_arguments, true), granularity_threshold: strategy_threshold(strategy) ) # Start event queue for shrinking {:ok, event_queue} = EventQueue.start_link() try do start_time = System.monotonic_time(:millisecond) # Perform shrinking on the already-shrunk sequence shrink_result = Shrinker.shrink(report.shrunk_sequence, failed_at_index: report.failed_at_index, failure_reason: report.failure_reason, model: model, adapter: adapter, adapter_config: adapter_config, config: shrinker_config, event_queue: event_queue ) # Re-execute to get fresh state {:ok, fresh_result} = Executor.run(shrink_result.sequence, model, adapter, adapter_config: adapter_config, event_queue: event_queue ) end_time = System.monotonic_time(:millisecond) # Create updated failure report new_report = FailureReport.new( seed: report.seed, run_number: report.run_number, original_sequence: report.original_sequence, shrunk_sequence: shrink_result.sequence, failed_at_index: fresh_result.failed_at_index, failure_reason: fresh_result.failure_reason, shrink_iterations: report.shrink_iterations + shrink_result.iterations, shrink_time_ms: report.shrink_time_ms + (end_time - start_time), event_log: fresh_result.event_log, projections: fresh_result.projections, projections_before: fresh_result.projections_before, refs: fresh_result.refs, model: model, adapter: adapter, linearization: fresh_result.linearization ) {:ok, new_report} after EventQueue.stop(event_queue) end end end # Strategy configuration helpers defp strategy_iterations(:quick, opts), do: Keyword.get(opts, :max_iterations, 500) defp strategy_iterations(:thorough, opts), do: Keyword.get(opts, :max_iterations, 2000) defp strategy_iterations(:exhaustive, opts), do: Keyword.get(opts, :max_iterations, 10000) defp strategy_time(:quick, opts), do: Keyword.get(opts, :max_time_ms, 10_000) defp strategy_time(:thorough, opts), do: Keyword.get(opts, :max_time_ms, 60_000) defp strategy_time(:exhaustive, opts), do: Keyword.get(opts, :max_time_ms, 300_000) defp strategy_threshold(:quick), do: 4 defp strategy_threshold(:thorough), do: 8 defp strategy_threshold(:exhaustive), do: 16 @doc """ Explain why each command in a failure's shrunk sequence is needed. Delegates to `PropertyDamage.Analysis.explain/1`. See that module for detailed documentation. """ @spec explain(FailureReport.t()) :: map() defdelegate explain(report), to: PropertyDamage.Analysis @doc """ Find the minimal change that eliminates the failure. Delegates to `PropertyDamage.Analysis.isolate_trigger/1`. See that module for detailed documentation. """ @spec isolate_trigger(FailureReport.t(), keyword()) :: {:ok, map()} | {:error, term()} defdelegate isolate_trigger(report, opts \\ []), to: PropertyDamage.Analysis @doc """ Generate a reproducible test case from a failure. Delegates to `PropertyDamage.Analysis.generate_test/2`. See that module for detailed documentation. """ @spec generate_test(FailureReport.t(), keyword()) :: String.t() defdelegate generate_test(report, opts \\ []), to: PropertyDamage.Analysis # ============================================================================ # Persistence API # ============================================================================ @doc """ Save a failure report to disk for later analysis or regression testing. ## Options - `:filename` - Custom filename (default: auto-generated from metadata) - `:overwrite` - Whether to overwrite existing files (default: false) ## Examples {:error, failure} = PropertyDamage.run(model: M, adapter: A) # Save with auto-generated name {:ok, path} = PropertyDamage.save_failure(failure, "failures/") # Save with custom name {:ok, path} = PropertyDamage.save_failure(failure, "failures/", filename: "currency-bug.pd") """ @spec save_failure(FailureReport.t(), Path.t(), keyword()) :: {:ok, Path.t()} | {:error, term()} defdelegate save_failure(report, directory, opts \\ []), to: PropertyDamage.Persistence, as: :save @doc """ Load a previously saved failure report. ## Examples {:ok, failure} = PropertyDamage.load_failure("failures/currency-bug.pd") PropertyDamage.replay(failure) """ @spec load_failure(Path.t()) :: {:ok, FailureReport.t()} | {:error, term()} defdelegate load_failure(path), to: PropertyDamage.Persistence, as: :load @doc """ List all saved failures in a directory. ## Options - `:sort` - Sort order: `:newest`, `:oldest`, `:seed` (default: `:newest`) - `:filter` - Filter function `(metadata -> boolean)` ## Examples failures = PropertyDamage.list_failures("failures/") # Only check failures failures = PropertyDamage.list_failures("failures/", filter: &(&1.failure_type == :check_failed)) """ @spec list_failures(Path.t(), keyword()) :: [map()] defdelegate list_failures(directory, opts \\ []), to: PropertyDamage.Persistence, as: :list @doc """ Delete a saved failure file. """ @spec delete_failure(Path.t()) :: :ok | {:error, term()} defdelegate delete_failure(path), to: PropertyDamage.Persistence, as: :delete # ============================================================================ # Replay API # ============================================================================ @doc """ Replay a failure sequence step-by-step for debugging. Executes each command in the shrunk sequence and returns detailed information about each step including events and projection states. ## Options - `:adapter_config` - Override adapter configuration - `:stop_on_failure` - Stop at first failure (default: true) ## Example {:error, failure} = PropertyDamage.run(model: M, adapter: A) {:ok, steps} = PropertyDamage.replay(failure) Enum.each(steps, fn step -> IO.puts("[\#{step.index}] \#{step.command_name}") IO.inspect(step.projections) end) For interactive stepping, use `PropertyDamage.Replay` directly: {:ok, session} = PropertyDamage.Replay.start(failure) {:ok, session, step} = PropertyDamage.Replay.step(session) """ @spec replay(FailureReport.t(), keyword()) :: {:ok, [PropertyDamage.Replay.step()]} | {:error, term()} defdelegate replay(failure, opts \\ []), to: PropertyDamage.Replay, as: :run # ============================================================================ # Seed Library API # ============================================================================ @doc """ Load a seed library from disk. Returns an empty library if the file doesn't exist. ## Example {:ok, library} = PropertyDamage.load_seed_library("seeds.json") """ @spec load_seed_library(Path.t()) :: {:ok, PropertyDamage.SeedLibrary.t()} | {:error, term()} defdelegate load_seed_library(path \\ "property_damage_seeds.json"), to: PropertyDamage.SeedLibrary, as: :load @doc """ Save a seed library to disk. ## Example :ok = PropertyDamage.save_seed_library(library, "seeds.json") """ @spec save_seed_library(PropertyDamage.SeedLibrary.t(), Path.t()) :: :ok | {:error, term()} defdelegate save_seed_library(library, path \\ "property_damage_seeds.json"), to: PropertyDamage.SeedLibrary, as: :save @doc """ Add a failure to the seed library. ## Options - `:tags` - Categorization tags (e.g., `[:currency, :race_condition]`) - `:description` - Human-readable description ## Example {:error, failure} = PropertyDamage.run(model: M, adapter: A) {:ok, library} = PropertyDamage.add_to_seed_library(library, failure, tags: [:currency_mismatch], description: "Capture with different currency than authorization" ) """ @spec add_to_seed_library(PropertyDamage.SeedLibrary.t(), FailureReport.t(), keyword()) :: {:ok, PropertyDamage.SeedLibrary.t()} | {:error, term()} defdelegate add_to_seed_library(library, failure, opts \\ []), to: PropertyDamage.SeedLibrary, as: :add # ============================================================================ # Coverage API # ============================================================================ @doc """ Get coverage statistics from a test result. ## Example result = PropertyDamage.run(model: M, adapter: A) coverage = PropertyDamage.coverage(result, M) IO.puts(PropertyDamage.Coverage.format(coverage)) """ @spec coverage({:ok, map()} | {:error, FailureReport.t()}, module()) :: PropertyDamage.Coverage.t() defdelegate coverage(result, model), to: PropertyDamage.Coverage, as: :from_result # ============================================================================ # Flakiness Detection API # ============================================================================ @doc """ Check if a seed produces deterministic results. Runs the same seed multiple times to detect non-deterministic behavior in the system under test. ## Options - `:runs` - Number of times to run (default: 5) - `:adapter_config` - Adapter configuration - `:max_commands` - Maximum commands per run (default: 50) - `:verbose` - Print progress (default: false) ## Returns - `{:ok, :deterministic}` - Same result every time - `{:ok, :flaky, stats}` - Different results, with statistics - `{:error, reason}` - Check failed ## Example case PropertyDamage.check_determinism(M, A, 512902757, runs: 10) do {:ok, :deterministic} -> IO.puts("Seed is deterministic") {:ok, :flaky, stats} -> IO.puts("FLAKY: passed \#{stats.passes}/\#{stats.runs} times") end """ @spec check_determinism(module(), module(), integer(), keyword()) :: PropertyDamage.Flakiness.result() defdelegate check_determinism(model, adapter, seed, opts \\ []), to: PropertyDamage.Flakiness, as: :check @doc """ Discover flaky seeds by testing random seeds. ## Options - `:num_seeds` - Number of random seeds to test (default: 10) - `:runs_per_seed` - Runs per seed (default: 3) - `:verbose` - Print progress (default: false) ## Returns List of `{seed, flaky_stats}` for seeds that are flaky. ## Example flaky_seeds = PropertyDamage.discover_flaky_seeds(M, A, num_seeds: 20) IO.puts("Found \#{length(flaky_seeds)} flaky seeds") """ @spec discover_flaky_seeds(module(), module(), keyword()) :: [{integer(), PropertyDamage.Flakiness.flaky_stats()}] defdelegate discover_flaky_seeds(model, adapter, opts \\ []), to: PropertyDamage.Flakiness, as: :discover_flaky # ============================================================================ # Assertion Helpers # ============================================================================ @doc """ Convenience function to fail an assertion with a message and optional data. Use this in projection assertions when you don't need a custom exception type. ## Examples # Simple failure PropertyDamage.fail!("balance is negative") # With context data PropertyDamage.fail!("balance is negative", balance: -50, account_id: "acc_123") # In a projection assertion @trigger every: 1 def assert_balance_positive(state, _cmd) do if state.balance < 0 do PropertyDamage.fail!("negative balance", balance: state.balance) end end ## Custom Exceptions For richer error context, define your own exception types: defmodule MyApp.BalanceViolation do defexception [:balance, :requirement] def message(%{balance: b}) do "Balance is negative: \#{b}" end end # Then raise directly: raise %MyApp.BalanceViolation{balance: -50, requirement: "REQ-001"} """ @spec fail!(String.t(), keyword()) :: no_return() def fail!(message, data \\ []) do raise %PropertyDamage.AssertionFailed{message: message, data: Map.new(data)} end @doc false defmacro __using__(_opts) do quote do import PropertyDamage, only: [] Module.register_attribute(__MODULE__, :property_damage_model, persist: true) Module.register_attribute(__MODULE__, :property_damage_adapter, persist: true) end end end