Automated test runner case for Arazzo documents

Basic Usage

Define use Cuerdo.ArazzoCase in your test module, and add arazzo_document_test macro referencing the document you want to test

defmodule MyArazzoTest do
  use Cuerdo.ArazzoCase

  arazzo_document_test document: YamlElixir.read_from_file!("spec/to/arazzo.yaml")
end

Filtering workflows

You can opt-in and opt-out from executing specific workflows via the :only and :exclude options respectively. Some examples

# Executes "workflow1" and "workflow2"
arazzo_document_test only: ["workflow1", "workflow2"], document: ...

# Executes every workflow defined in the document except for "workflow1"
arazzo_document_test exclude: ["workflow1"], document: ...

# Executes "workflow1"
arazzo_document_test only: ["workflow1", "workflow2"], exclude: ["workflow2"], document: ...

Fine-tuning input

Consider for example a "book" input containing the book's title, author and ISBN

  {
    "type": "object",
    "additionalProperties": false,
    "properties": {
      "title": {"type": "string", "minLength": 1},
      "authorName": {"type": "string", "minLength": 1},
      "isbn": {"type": "string", "pattern": "^97(8|9)[0-9]{10}$"}
    }
  }

ISBNs consist of 13 digits, where the last digit is a check digit. In order for an ISBN to be valid, the total weighted sum must be a multiple of 10. If the service under test checks that the ISBN is valid then tests will fail because JSON Schema cannot express this constraint and invalid ISBNs will be generated.

We can work around this issue via the :transform_inputs option.

First, let's define a function that generates valid ISBN identifiers

defmodule MyModule do
  def valid_isbn do
    StreamData.bind(MoreStreamData.from_regex("^97(8|9)[0-9]{10}$"), fn invalid_isbn ->
      {digits, _wrong_check_digit} = String.split_at(invalid_isbn, 12)
      StreamData.constant(digits <> to_string(check_digit(digits)))
    end)
  end

  defp check_digit(digits) do
    digits
    |> String.codepoints()
    |> Enum.with_index()
    |> Enum.sum_by(fn {digit, idx} ->
      digit = String.to_integer(digit)
      if(rem(idx, 2) == 0, do: digit, else: 3 * digit)
    end)
    |> then(fn total -> 10 - rem(total, 10) end)
  end

Then we will define a function that receives a book input and inserts the valid ISBN

def with_valid_isbn(book) do
  StreamData.bind(valid_isbn(), fn isbn ->
    StreamData.constant(Map.put(book, "isbn", isbn))
  end)
end

Finally, in the arazzo_document_test definition we have to add :transform_inputs option with the name of the workflow and the function as {Module, :function_name, [args]}. The generated value will be passed through the function we defineds, ensuring that we test the workflow with valid data.

defmodule MyModuleTest do
  use Cuerdo.ArazzoCase

  arazzo_document_test transform_inputs: %{
    "createBookWorkflow" => {MyModule, :with_valid_isbn, []}
  },
  document: ...
end

Multiple arazzo_document_test

You can define multipel arazzo_document_test in the same module. This is useful if, for example, you have an expensive workflow that you want to run fewer times, or if you want to run workflows both with default generated inputs and custom inputs

defmodule MyArazzoTest do
  use Cuerdo.ArazzoCase

  arazzo_document_test max_runs: 20,
                       exclude: ["expensiveWorkflowId"],
                       document: YamlElixir.read_from_file!("path/to/arazzo.yaml")

  arazzo_document_test max_runs: 3,
                       only: ["expensiveWorkflowId"],
                       document: YamlElixir.read_from_file!("path/to/arazzo.yaml")
end

Keep in mind that the test names are generated based on each workflow's workflowId field. If you run the same workflows with different inputs you must use the :prefix option, otherwise the test generation step will fail

defmodule MyArazzoTest do
  use Cuerdo.ArazzoCase

  arazzo_document_test prefix: "custom_inputs",
                       transform_inputs: %{"workflowId" => {Module, :function, []}},
                       document: YamlElixir.read_from_file!("path/to/arazzo.yaml")

  arazzo_document_test prefix: "default_inputs,
                       document: YamlElixir.read_from_file!("path/to/arazzo.yaml")
end