PropertyDamage.FailureIntelligence (PropertyDamage v0.2.0)

View Source

Failure Intelligence - Pattern detection and fix verification for failures.

This module provides tools for:

  • Fingerprinting: Extract comparable features from failures
  • Similarity: Compare failures and identify duplicates
  • Pattern Detection: Cluster similar failures to find root causes
  • Fix Verification: Confirm fixes are robust with seed variations

Quick Start

# Analyze a set of failures
failures = [failure1, failure2, failure3]
analysis = PropertyDamage.FailureIntelligence.analyze(failures)

# Check if two failures are similar
similar? = PropertyDamage.FailureIntelligence.similar?(failure1, failure2)

# Verify a fix is robust
result = PropertyDamage.FailureIntelligence.verify_fix(failure, MyModel,
  adapter: MyAdapter,
  adapter_config: %{}
)

Pattern Detection

The system uses fingerprinting to extract key characteristics:

  • Failure type (check failure, exception, timeout, etc.)
  • Check name that failed
  • Command type that triggered the failure
  • Event types produced
  • Error category and pattern

Similar failures are clustered together to identify systemic issues.

Fix Verification

When you believe a bug is fixed, verify it:

result = PropertyDamage.FailureIntelligence.verify_fix(failure, MyModel,
  adapter: MyAdapter,
  max_variations: 20
)

case result.status do
  :verified -> "Fix confirmed with high confidence"
  :still_failing -> "Original failure still reproduces"
  :partially_fixed -> "Some variations still fail"
  :flaky -> "Intermittent failures detected"
end

Summary

Functions

Analyzes a set of failures to identify patterns.

Clusters failures by similarity.

Performs a detailed comparison between two failures.

Finds potential duplicate failures (similarity > 0.90).

Finds failures similar to the target from a list.

Creates a fingerprint from a failure report.

Returns a short hash of a fingerprint for display.

Formats a verification result for display.

Groups failures by their fingerprint hash for quick deduplication.

Checks if a failure matches an existing pattern cluster.

Checks if two failures are similar (score >= 0.70).

Checks if two failures are similar using a custom threshold.

Computes the similarity score between two failures.

Quick check if a single seed still fails.

Returns a summary of failure intelligence analysis.

Verifies that a fix is robust by testing the original seed and variations.

Verifies multiple fixes at once.

Types

analysis()

verification_result()

Functions

analyze(failures, opts \\ [])

@spec analyze(
  [PropertyDamage.FailureReport.t()],
  keyword()
) :: analysis()

Analyzes a set of failures to identify patterns.

Returns clustering information and pattern summaries.

Options

  • :threshold - Similarity threshold for clustering (default: 0.70)
  • :min_cluster_size - Minimum size for a pattern cluster (default: 2)

Example

analysis = PropertyDamage.FailureIntelligence.analyze(failures)

IO.puts(analysis.pattern_summary)
# => "Analyzed 15 failures:
#     - 3 distinct patterns (12 failures)
#     - 3 unique failures (no pattern match)
#
#     Top patterns:
#       - Check failure in :balance_valid during DebitAccount (5 occurrences)
#       - Invariant violation during CreditAccount (4 occurrences)"

# Get specific patterns
Enum.each(analysis.clusters, fn cluster ->
  IO.puts("Pattern: #{cluster.pattern.description}")
  IO.puts("Occurrences: #{cluster.size}")
end)

cluster(failures, opts \\ [])

Clusters failures by similarity.

Returns a list of clusters, each containing similar failures.

Example

clusters = PropertyDamage.FailureIntelligence.cluster(failures)
Enum.each(clusters, fn c ->
  IO.puts("Cluster #{c.id}: #{c.size} failures")
end)

compare(f1, f2)

Performs a detailed comparison between two failures.

Returns the overall score, per-component breakdown, and similarity determination.

Example

comparison = PropertyDamage.FailureIntelligence.compare(failure1, failure2)
# => %{
#   score: 0.85,
#   breakdown: %{failure_type: 1.0, check_name: 1.0, command_type: 0.8, ...},
#   is_similar: true
# }

find_duplicates(failures)

Finds potential duplicate failures (similarity > 0.90).

Example

dupes = PropertyDamage.FailureIntelligence.find_duplicates(failures)
Enum.each(dupes, fn {f1, f2, score} ->
  IO.puts("Seeds #{f1.seed} and #{f2.seed} are #{score * 100}% similar")
end)

find_similar(target, failures, opts \\ [])

Finds failures similar to the target from a list.

Options

  • :threshold - Minimum similarity score (default: 0.70)
  • :limit - Maximum number of results (default: unlimited)

Example

similar = PropertyDamage.FailureIntelligence.find_similar(failure, all_failures,
  threshold: 0.80,
  limit: 5
)
# => [{similar_failure1, 0.92}, {similar_failure2, 0.85}, ...]

fingerprint(failure)

Creates a fingerprint from a failure report.

A fingerprint captures the essential characteristics of a failure for comparison.

Example

fingerprint = PropertyDamage.FailureIntelligence.fingerprint(failure)
# => %Fingerprint{failure_type: :check_failed, check_name: :balance_valid, ...}

fingerprint_hash(failure)

@spec fingerprint_hash(PropertyDamage.FailureReport.t()) :: String.t()

Returns a short hash of a fingerprint for display.

Example

hash = PropertyDamage.FailureIntelligence.fingerprint_hash(failure)
# => "a1b2c3d4"

format_verification(result)

@spec format_verification(verification_result()) :: String.t()

Formats a verification result for display.

Example

result = PropertyDamage.FailureIntelligence.verify_fix(failure, model, opts)
IO.puts(PropertyDamage.FailureIntelligence.format_verification(result))

group_by_fingerprint(failures)

@spec group_by_fingerprint([PropertyDamage.FailureReport.t()]) :: %{
  required(String.t()) => [PropertyDamage.FailureReport.t()]
}

Groups failures by their fingerprint hash for quick deduplication.

Example

groups = PropertyDamage.FailureIntelligence.group_by_fingerprint(failures)
Enum.each(groups, fn {hash, group} ->
  IO.puts("Hash #{hash}: #{length(group)} failures")
end)

match_pattern(failure, clusters, opts \\ [])

Checks if a failure matches an existing pattern cluster.

Example

case PropertyDamage.FailureIntelligence.match_pattern(new_failure, clusters) do
  nil -> IO.puts("New unique failure")
  cluster -> IO.puts("Matches pattern: #{cluster.pattern.description}")
end

similar?(f1, f2)

Checks if two failures are similar (score >= 0.70).

Example

if PropertyDamage.FailureIntelligence.similar?(failure1, failure2) do
  IO.puts("These failures likely have the same root cause")
end

similar?(f1, f2, threshold)

Checks if two failures are similar using a custom threshold.

similarity_score(f1, f2)

Computes the similarity score between two failures.

Returns a score between 0.0 (completely different) and 1.0 (identical).

Example

score = PropertyDamage.FailureIntelligence.similarity_score(failure1, failure2)
# => 0.85

still_fails?(seed, model, adapter, adapter_config \\ %{})

@spec still_fails?(integer(), module(), module(), map()) :: boolean()

Quick check if a single seed still fails.

Example

if PropertyDamage.FailureIntelligence.still_fails?(12345, MyModel, MyAdapter) do
  IO.puts("Bug not fixed yet!")
end

summary(failures)

@spec summary([PropertyDamage.FailureReport.t()]) :: String.t()

Returns a summary of failure intelligence analysis.

Combines pattern detection with fix suggestions.

Example

summary = PropertyDamage.FailureIntelligence.summary(failures)
IO.puts(summary)

verify_fix(failure, model, opts \\ [])

Verifies that a fix is robust by testing the original seed and variations.

This helps ensure that:

  1. The original failure no longer reproduces
  2. Similar command sequences also pass
  3. The fix is robust across different conditions

Options

  • :adapter - The adapter module to use (required)
  • :adapter_config - Configuration for the adapter
  • :max_variations - Maximum number of seed variations to test (default: 10)
  • :variation_range - Range for generating seed variations (default: 1000)

Example

result = PropertyDamage.FailureIntelligence.verify_fix(failure, MyModel,
  adapter: MyAdapter,
  adapter_config: %{base_url: "http://localhost:4000"},
  max_variations: 20
)

case result.status do
  :verified ->
    IO.puts("Fix verified with #{result.confidence * 100}% confidence")

  :still_failing ->
    IO.puts("Original failure still reproduces!")

  :partially_fixed ->
    IO.puts("Fix incomplete. #{result.variations_failed} variations still fail")

  :flaky ->
    IO.puts("Intermittent failures detected. May be timing-related.")
end

Return Value

%{
  status: :verified | :still_failing | :partially_fixed | :flaky,
  original_seed: 12345,
  original_passes: true,
  variations_run: 20,
  variations_passed: 18,
  variations_failed: 2,
  failed_variations: [12346, 12400],
  confidence: 0.95,
  summary: "Fix verified! Original seed and all 18 variations pass."
}

verify_fixes(failures, model, opts \\ [])

Verifies multiple fixes at once.

Useful for batch verification after a series of bug fixes.

Example

results = PropertyDamage.FailureIntelligence.verify_fixes(failures, MyModel,
  adapter: MyAdapter
)

Enum.each(results, fn {failure, result} ->
  IO.puts("Seed #{failure.seed}: #{result.status}")
end)