๐Ÿงช Eliot Testing Guide ๐Ÿงช

View Source

Welcome to the Eliot testing documentation! This guide explains how to run tests, understand the test suite, and create new tests for the Eliot system.

๐Ÿ“‹ Table of Contents

Test Suite Overview

Eliot includes a comprehensive test suite that validates:

  • Core Functionality: Correctness of module logic and public APIs.
  • OTP Behavior: Proper GenServer/Supervisor initialization, state changes, and shutdown.
  • Error Handling: Circuit breaker and retry logic correctness.
  • Integration: How different components of the system interact.

Tests are organized into two main categories:

  • Unit Tests: Testing individual modules and functions in isolation.
  • Integration Tests: Testing the interaction between different components (e.g., ErrorHandler and Logger).

Running Tests

Running All Tests

The simplest way to run all tests is using the standard mix command:

mix test

This will compile the project if needed and run all tests found in the test/ directory.

Running a Single Test File

To run tests from a single file:

mix test test/eliot/error_handler_test.exs

Running a Single Test

To run a specific test case from a file, you can specify the line number:

mix test test/eliot/error_handler_test.exs:25

Excluding or Including Tests with Tags

You can tag tests to run specific categories. For example, integration tests are tagged with :integration.

# Run only integration tests
mix test --only integration

# Run all tests except integration tests
mix test --exclude integration

Understanding Test Types

Unit Tests

Unit tests check individual modules and functions in isolation. These tests focus on:

  • Public function return values for given inputs.
  • GenServer callbacks and state transitions.
  • Logic inside private functions (tested via their public-facing callers).

Sample unit test files:

  • test/eliot/elixir_test.exs
  • test/eliot/error_handler_test.exs
  • test/eliot/logger_test.exs

Integration Tests

Integration tests verify that different components work together correctly. For Eliot, this includes:

  • Ensuring an error handled by ErrorHandler is correctly logged by Eliot.Logger.
  • Verifying the Eliot.Application correctly starts and supervises its children.

Sample integration test files:

  • test/eliot/integration/mqtt_integration_test.exs

Test Framework (ExUnit)

Eliot uses Elixir's built-in testing framework, ExUnit.

Test Structure

Tests are defined inside modules using use ExUnit.Case. The describe block groups related tests, and each test block defines a single test case.

defmodule Eliot.MyModuleTest do
  use ExUnit.Case, async: true

  describe "my_function/2" do
    test "does something correctly" do
      assert Eliot.MyModule.my_function(:a, :b) == :ok
    end
  end
end

Setup

You can use setup or setup_all blocks to run setup code before tests. setup runs before each test in a block, while setup_all runs once for the entire block.

defmodule Eliot.MyModuleTest do
  use ExUnit.Case, async: true

  setup do
    # This runs before each test
    {:ok, pid} = MyGenServer.start_link()
    # The returned value is passed to the test context
    %{pid: pid}
  end

  test "sends a message to the genserver", context do
    assert :ok == MyGenServer.do_something(context.pid)
  end
end

Assertions

ExUnit provides various assertion macros:

  • assert some_condition == true
  • refute some_condition == true
  • assert_raise(MyError, fn -> ... end)
  • assert_receive(:my_message, 100)
  • assert_received(:my_message)

Mocking

For isolating components from their dependencies, we recommend using a library like Mox. Mox allows you to define mock implementations of your modules' behaviors for testing.

Writing New Tests

To create a new unit test:

  1. Create a new file in the test/eliot/ directory, ending with _test.exs.
  2. Follow this basic structure:
    defmodule Eliot.NewFeatureTest do
    use ExUnit.Case, async: true
    
    describe "new_feature_function/1" do
     test "handles valid input correctly" do
       # Given valid input
       input = "valid"
       
       # When the function is called
       result = Eliot.NewFeature.new_feature_function(input)
       
       # Then the result is correct
       assert result == :ok
     end
    
     test "handles invalid input with an error tuple" do
       assert Eliot.NewFeature.new_feature_function(nil) == {:error, :invalid_input}
     end
    end
    end
  3. Run mix test to ensure your new tests are picked up and pass.

Test Coverage

We aim for high code coverage across the codebase.

Measuring Coverage

You can generate a test coverage report by running:

mix test --cover

This generates a report in cover/index.html which you can open in your browser to see which lines of code are covered by tests.

Coverage Requirements

  • Core functionality in Eliot, Eliot.ErrorHandler, and Eliot.Logger should have >95% coverage.
  • All public API functions must have tests.
  • Error handling paths must be tested explicitly.

Continuous Integration

Eliot uses GitHub Actions for continuous integration. The workflow runs automatically on pull requests and includes:

  • Running mix test
  • Checking formatting with mix format --check-formatted
  • Running static analysis with mix credo

Testing Best Practices

  1. Test One Thing at a Time: Each test should verify a single, specific behavior.
  2. Use Descriptive Names: Test names should clearly describe what is being tested.
  3. Isolate Tests: Use async: true where possible to run tests concurrently and ensure they don't depend on shared state.
  4. Test the Public API: Focus tests on the public-facing functions of your modules, not the private implementation details.
  5. Test Failure Cases: Verify that your functions fail appropriately with invalid inputs and that your application recovers gracefully from errors.

If you have questions about testing, please reach out via GitHub issues.