PropertyDamage.Regression (PropertyDamage v0.2.0)

View Source

Automatic regression test management for PropertyDamage.

When PropertyDamage discovers a failure, this module can automatically:

  • Save failure files for later analysis
  • Add seeds to a seed library for regression testing
  • Generate ExUnit tests for CI/CD
  • Deduplicate similar failures to avoid noise

Usage with run/1

The simplest way to use regression management is via the :regression option:

PropertyDamage.run(
  model: MyModel,
  adapter: MyAdapter,
  regression: [
    save_failures: "failures/",
    seed_library: "seeds.json",
    generate_tests: "test/regressions/",
    tags: [:auto_detected],
    dedup: true
  ]
)

Using on_failure Callback

You can also use the handler/1 function with :on_failure:

PropertyDamage.run(
  model: MyModel,
  adapter: MyAdapter,
  on_failure: PropertyDamage.Regression.handler([
    save_failures: "failures/",
    seed_library: "seeds.json"
  ])
)

Composing Handlers

For custom behavior, compose multiple handlers:

PropertyDamage.run(
  model: MyModel,
  adapter: MyAdapter,
  on_failure: PropertyDamage.Regression.compose([
    PropertyDamage.Regression.save_failure("failures/"),
    PropertyDamage.Regression.add_to_library("seeds.json"),
    fn report -> Logger.warning("Failure: #{report.seed}") end
  ])
)

Deduplication

When dedup: true is set, failures are checked against existing failures before being added. This prevents noise from multiple runs finding the same underlying bug.

regression: [
  save_failures: "failures/",
  dedup: true,
  dedup_threshold: 0.90  # 90% similarity threshold
]

Summary

Types

handler()
regression_opts()

Functions

add_to_library(path, opts \\ [])

Creates a handler that adds failures to a seed library.

batch_summary(results)

Generates a summary report for batch processing results.

check_duplicate(failure, opts)

Checks if a failure is a duplicate of an existing failure.

compose(handlers)

Composes multiple handlers into a single handler.

find_duplicate(failure, existing, threshold)

Finds a duplicate failure from a list.

format_batch_summary(summary)

Formats batch summary for display.

generate_test(directory, opts \\ [])

Creates a handler that generates ExUnit regression tests.

handle_failure(failure, opts \\ [])

Processes a failure according to regression options.

handler(opts \\ [])

Creates a handler function from regression options.

process_batch(failures, opts \\ [])

Processes multiple failures, deduplicating across the batch.

save_failure(directory, opts \\ [])

Creates a handler that saves failures to a directory.

Types

handler()

@type handler() :: (PropertyDamage.FailureReport.t() -> any())

regression_opts()

@type regression_opts() :: [
  save_failures: Path.t(),
  seed_library: Path.t(),
  generate_tests: Path.t(),
  tags: [atom()],
  description: String.t() | nil,
  dedup: boolean(),
  dedup_threshold: float(),
  dedup_source: :failures,
  verbose: boolean(),
  adapter: module() | nil,
  base_url: String.t() | nil
]

Functions

add_to_library(path, opts \\ [])

@spec add_to_library(
  Path.t(),
  keyword()
) :: handler()

Creates a handler that adds failures to a seed library.

Options

  • :tags - Tags to add to the entry (default: [:auto_detected])
  • :description - Optional description

Example

PropertyDamage.run(
  model: M,
  adapter: A,
  on_failure: PropertyDamage.Regression.add_to_library("seeds.json",
    tags: [:balance_bug]
  )
)

batch_summary(results)

@spec batch_summary([map()]) :: map()

Generates a summary report for batch processing results.

check_duplicate(failure, opts)

@spec check_duplicate(
  PropertyDamage.FailureReport.t(),
  keyword()
) :: {boolean(), term()}

Checks if a failure is a duplicate of an existing failure.

Returns {true, reason} if duplicate, {false, nil} otherwise.

Options

  • :dedup_threshold - Similarity threshold (default: 0.90)
  • :save_failures - Directory containing saved failures to compare against

compose(handlers)

@spec compose([handler()]) :: handler()

Composes multiple handlers into a single handler.

All handlers are called in order. Errors in one handler don't prevent subsequent handlers from running.

Example

PropertyDamage.run(
  model: M,
  adapter: A,
  on_failure: PropertyDamage.Regression.compose([
    PropertyDamage.Regression.save_failure("failures/"),
    PropertyDamage.Regression.add_to_library("seeds.json"),
    fn report -> IO.puts("Found: #{report.seed}") end
  ])
)

find_duplicate(failure, existing, threshold)

@spec find_duplicate(
  PropertyDamage.FailureReport.t(),
  [PropertyDamage.FailureReport.t()],
  float()
) ::
  {PropertyDamage.FailureReport.t(), float()} | nil

Finds a duplicate failure from a list.

Returns {similar_failure, score} if found, nil otherwise.

format_batch_summary(summary)

@spec format_batch_summary(map()) :: String.t()

Formats batch summary for display.

generate_test(directory, opts \\ [])

@spec generate_test(
  Path.t(),
  keyword()
) :: handler()

Creates a handler that generates ExUnit regression tests.

Options

  • :adapter - Adapter module (for HTTP spec generation)
  • :base_url - Base URL for HTTP calls

Example

PropertyDamage.run(
  model: M,
  adapter: A,
  on_failure: PropertyDamage.Regression.generate_test("test/regressions/",
    adapter: MyHTTPAdapter
  )
)

handle_failure(failure, opts \\ [])

@spec handle_failure(PropertyDamage.FailureReport.t(), regression_opts()) :: map()

Processes a failure according to regression options.

Returns a summary of actions taken.

Example

result = PropertyDamage.Regression.handle_failure(failure, [
  save_failures: "failures/",
  seed_library: "seeds.json"
])

# => %{
#   saved_failure: {:ok, "failures/..."},
#   added_to_library: {:ok, "seeds.json"},
#   generated_test: nil,
#   skipped: false,
#   skip_reason: nil
# }

handler(opts \\ [])

@spec handler(regression_opts()) :: handler()

Creates a handler function from regression options.

This is the main entry point for creating regression handlers. The returned function can be passed to :on_failure in PropertyDamage.run/1.

Options

  • :save_failures - Directory to save failure files
  • :seed_library - Path to seed library JSON file
  • :generate_tests - Directory to generate ExUnit test files
  • :tags - Tags to add to seed library entries (default: [:auto_detected])
  • :description - Optional description for seed library entries
  • :dedup - Enable deduplication (default: false)
  • :dedup_threshold - Similarity threshold for dedup (default: 0.90)
  • :dedup_source - Where to check for duplicates. Only :failures (saved failure files) is supported.
  • :verbose - Print actions taken (default: false)
  • :adapter - Adapter module for script generation (required for generate_tests with scripts)
  • :base_url - Base URL for script generation

Example

handler = PropertyDamage.Regression.handler(
  save_failures: "failures/",
  seed_library: "seeds.json",
  dedup: true,
  verbose: true
)

PropertyDamage.run(model: M, adapter: A, on_failure: handler)

process_batch(failures, opts \\ [])

@spec process_batch([PropertyDamage.FailureReport.t()], regression_opts()) :: [map()]

Processes multiple failures, deduplicating across the batch.

Useful when you have accumulated failures and want to add only unique ones.

Example

failures = [failure1, failure2, failure3]
results = PropertyDamage.Regression.process_batch(failures, [
  seed_library: "seeds.json",
  dedup: true
])

save_failure(directory, opts \\ [])

@spec save_failure(
  Path.t(),
  keyword()
) :: handler()

Creates a handler that saves failures to a directory.

Example

PropertyDamage.run(
  model: M,
  adapter: A,
  on_failure: PropertyDamage.Regression.save_failure("failures/")
)