LivebookTest.Runner (livebook_test v0.1.0)

Copy Markdown View Source

Executes exported Elixir scripts and collects results.

This is the fourth stage in the LivebookTest pipeline:

Discovery → Exporter → DependencyPatcher → **Runner** → Report

The runner executes each .exs script as a subprocess using System.cmd/3, collecting:

  • Exit status (0 = success, non-zero = failure)
  • Captured stdout
  • Captured stderr (merged into stdout via stderr_to_stdout: true)
  • Execution timing

Why subprocess execution?

Livebook notebooks often call Mix.install/2, which modifies the global application environment. Running scripts in isolated subprocesses prevents side effects from leaking between notebook tests.

Note on timeouts: When a notebook execution times out, the BEAM task is shut down via Task.shutdown/2. However, the underlying elixir OS process may continue running (e.g., during a lengthy Mix.install compilation). This is a known limitation of System.cmd-based subprocess management. If this becomes an issue, consider configuring a shorter timeout or using a process supervision strategy that tracks OS process lifecycles.

Examples

iex> {:ok, script_path} = LivebookTest.Exporter.to_temp_file("examples/basic.livemd")
iex> {:ok, result} = LivebookTest.Runner.run(script_path)
iex> result.exit_status
0

Summary

Types

run_outcome()

Outcome of running a single script

run_result()

Result of a single notebook execution

Functions

run(script_path, opts \\ [])

Runs an Elixir script and collects execution results.

run_all(script_pairs, opts \\ [])

Runs multiple scripts sequentially, collecting all results.

success?(runner)

Returns whether a run result indicates success.

Types

run_outcome()

@type run_outcome() :: {:ok, run_result()} | {:error, term()}

Outcome of running a single script

run_result()

@type run_result() :: %LivebookTest.Runner{
  duration_ms: non_neg_integer(),
  exit_status: non_neg_integer(),
  notebook_path: Path.t(),
  script_path: Path.t(),
  stderr: String.t(),
  stdout: String.t(),
  timed_out: boolean()
}

Result of a single notebook execution

Functions

run(script_path, opts \\ [])

@spec run(
  Path.t(),
  keyword()
) :: run_outcome()

Runs an Elixir script and collects execution results.

Executes elixir <script_path> as a subprocess with the given timeout. Returns a structured result with exit status, output, and timing information.

Examples

iex> {:ok, script_path} = LivebookTest.Exporter.to_temp_file("examples/basic.livemd")
iex> {:ok, result} = LivebookTest.Runner.run(script_path)
iex> is_integer(result.exit_status)
true

run_all(script_pairs, opts \\ [])

@spec run_all(
  [{Path.t(), Path.t()}],
  keyword()
) :: [run_result()]

Runs multiple scripts sequentially, collecting all results.

Returns a list of results in the same order as the input scripts.

Examples

iex> paths = []
iex> results = LivebookTest.Runner.run_all(paths)
iex> results
[]

success?(runner)

@spec success?(run_result()) :: boolean()

Returns whether a run result indicates success.

Examples

iex> result = %LivebookTest.Runner{notebook_path: "a.livemd", script_path: "a.exs", exit_status: 0, stdout: "", stderr: "", duration_ms: 100, timed_out: false}
iex> LivebookTest.Runner.success?(result)
true

iex> result = %LivebookTest.Runner{notebook_path: "a.livemd", script_path: "a.exs", exit_status: 1, stdout: "", stderr: "error", duration_ms: 100, timed_out: false}
iex> LivebookTest.Runner.success?(result)
false