livebook_test

mix test for Livebooks - Keep your Livebook examples honest.

livebook_test brings mix test-style workflows to Livebook notebooks. It discovers .livemd files, converts them to executable Elixir scripts, runs them, and reports failures - locally and in CI/CD.

Why?

Livebook notebooks are great for examples, tutorials, and interactive documentation. But they drift:

• A dependency update breaks your example notebook
• A refactoring breaks the API shown in a tutorial
• Published notebooks reference old package versions

livebook_test catches these problems before your users do.

Installation

Add to your Mix project:

def deps do
  [
    {:livebook_test, "~> 0.1.0", only: [:dev, :test], runtime: false}
  ]
end

Then fetch dependencies:

mix deps.get

Quick Start

# Run all discovered notebooks
mix livebook.test

# Run with verbose output
mix livebook.test --verbose

# Test against local checkout instead of Hex
mix livebook.test --mode local

# Run a specific notebook
mix livebook.test --path examples/basic.livemd

Example Notebooks

Configuration

Configure in config/config.exs:

config :livebook_test,
  paths: ["livebooks/**/*.livemd", "examples/**/*.livemd"],
  exclude: ["**/broken/**/*.livemd"],
  dependency_mode: :remote,
  timeout: 60_000,
  local_deps: [],
  verbose: false

Options

Local Dependency Testing

The killer feature: notebooks that use Mix.install can be automatically patched to use your local checkout.

Problem

Your example notebook says:

Mix.install([
  {:my_lib, "~> 0.5"}
])

But you want CI to test against the current checkout, not the published Hex version.

Solution

# config/config.exs
config :livebook_test,
  dependency_mode: :local,
  local_deps: [
    my_lib: "."
  ]

Now {:my_lib, "~> 0.5"} becomes {:my_lib, path: "."}.

Or via CLI:

mix livebook.test --mode local

CI/CD Integration

Add to your GitHub Actions workflow:

- name: Test Livebooks
  run: mix livebook.test

With local deps:

- name: Test Livebooks (local)
  run: mix livebook.test --mode local

The task exits with code 0 on success, 1 on failure, and 2 if no notebooks are discovered, perfect for CI gates.

CLI Options

mix livebook.test [options]

Options:
  --path PATTERN    Glob pattern for discovery (repeatable)
  --exclude PATTERN Glob pattern to exclude from discovery (repeatable)
  --mode MODE       Dependency mode: local or remote
  --timeout SECS    Per-notebook timeout in seconds
  --verbose         Show per-notebook details

Programmatic API

# Run with defaults
LivebookTest.run()

# Run with options
LivebookTest.run(paths: ["examples/**/*.livemd"], mode: :local, timeout: 120_000)

# Run and print report, returns exit code
LivebookTest.run_and_report(verbose: true)

Pipeline

Notebooks flow through a pipeline:

1. Discovery - Find .livemd files via glob patterns
2. Export - Convert to .exs scripts using Livebook.live_markdown_to_elixir/1
3. Patch - Optionally rewrite Mix.install deps to local paths
4. Run - Execute each script as an isolated subprocess
5. Report - Summarize results with pass/fail counts and timing

Example Output

3 notebooks
3 passed
0 failed
Total time: 2.1s

All notebooks passed! ✅

With failures:

3 notebooks
2 passed
1 failed
Total time: 5.3s

Failed notebooks:
--------------------

  examples/broken/broken.livemd
  exit: 1
    stderr:
    ** (RuntimeError) Intentional failure for testing

Architecture

Roadmap

License

MIT