Search documentation of Metastatic

# Supplemental Modules Guide

View Source

This guide explains Metastatic's supplemental module system - a mechanism for extending MetaAST with language-specific library integrations.

## Table of Contents

- Overview
- Core Concepts
- Architecture
- Using Supplementals
- Available Supplementals
- Creating Supplementals
- Best Practices
- API Reference

## Overview

Supplemental modules bridge the gap between MetaAST's language-agnostic representation and real-world library ecosystems. They enable:

- Cross-language transformations - Express actor patterns in Python/Erlang/JavaScript
- Library integrations - Map MetaAST constructs to third-party APIs (Pykka, asyncio, etc.)
- Semantic preservation - Maintain intent while adapting to language-specific idioms

### Example

    # MetaAST representing an actor call
    ast = {:actor_call, {:variable, "worker"}, "process", [data]}

    # Transform using Pykka supplemental for Python
    {:ok, python_ast} = Metastatic.Supplemental.Transformer.transform(ast, :python)
    # Result: {:function_call, "worker.ask", ["process", data]}

    # Same AST, different supplemental for Erlang
    {:ok, erlang_ast} = Metastatic.Supplemental.Transformer.transform(ast, :erlang)
    # Result: {:function_call, "gen_server:call", [worker, {:process, data}]}

## Core Concepts

### M2 Layer Extension

Supplementals operate at the M2 meta-model level, adding optional constructs beyond Core/Extended/Native:

    M2 Core    - Universals (literals, variables, operators, conditionals)
    M2 Extended - Common patterns (loops, lambdas, collections)
    M2 Native  - Language escapes
    M2 Supplemental - Library-specific extensions (actors, async, etc.)

Supplemental constructs are not part of the base MetaAST grammar - they're opt-in extensions that require explicit transformation.

### Semantic Contract

Each supplemental defines:

- Constructs - Which MetaAST node types it handles (e.g., `:actor_call`, `:async_await`)
- Target language - Single language it generates code for (`:python`, `:javascript`, etc.)
- Dependencies - External libraries required (`"pykka >= 3.0"`, `"asyncio"`)
- Transformation - How to convert supplemental constructs to concrete code

### Registry System

A centralized GenServer registry maintains all available supplementals, enabling:

- Fast lookup by construct type
- Language compatibility checking
- Conflict detection between competing supplementals
- Runtime registration/deregistration

## Architecture

### Components

    flowchart TD
        subgraph System["Supplemental System"]
            Behaviour["Behaviour<br/>(spec)"] --> Info["Info<br/>(metadata)"]
            Behaviour --> Registry

            Registry["Registry (GenServer)<br/>- by_construct index<br/>- by_language index<br/>- conflict detection"]
            Registry --> Transformer

            Transformer["Transformer<br/>- lookup + invoke supplementals<br/>- error handling"]
            Transformer --> Concrete

            Concrete["Concrete Supplementals<br/>- Python.Pykka<br/>- Python.Asyncio<br/>- ... (extensible)"]
        end

### Data Flow

    flowchart TD
        A["User AST with supplemental constructs"] --> B["Validator"]
        B --> C["Identify required supplementals"]
        C --> D["Registry lookup"]
        D --> E["Transformer"]
        E --> F["Invoke supplemental.transform/3"]
        F --> G["Concrete language AST"]

## Using Supplementals

### Basic Usage

    alias Metastatic.Supplemental.Transformer

    # Single construct transformation
    ast = {:actor_call, {:variable, "worker"}, "process", [data]}
    {:ok, result} = Transformer.transform(ast, :python)

    # Check availability before transforming
    if Transformer.available?(:actor_call, :python) do
      {:ok, transformed} = Transformer.transform(ast, :python)
    end

    # Get all supported constructs for a language
    constructs = Transformer.supported_constructs(:python)
    # => [:actor_call, :actor_cast, :spawn_actor, :async_await, :async_context, :gather]

### With Adapters

Supplementals integrate automatically when using the adapter pipeline:

    alias Metastatic.Builder

    # Source with supplemental constructs
    source = """
    actor = spawn_actor(Worker, [config])
    result = actor_call(actor, "process", [data])
    """

    # Adapter automatically uses supplementals during transformation
    {:ok, doc} = Builder.from_source(source, :python)

    # Round-trip preserves supplemental transformations
    {:ok, output} = Builder.to_source(doc)

### Configuration

Configure auto-registration in `config/config.exs`:

    config :metastatic, :supplementals,
      auto_register: [
        Metastatic.Supplemental.Python.Pykka,
        Metastatic.Supplemental.Python.Asyncio
      ]

### Validation

Use the validator to detect required supplementals in a document:

    alias Metastatic.{Document, Supplemental.Validator}

    doc = Document.new(ast, :python)
    {:ok, analysis} = Validator.validate(doc)

    # Check what supplementals are needed
    analysis.required_supplementals
    # => [:pykka, :asyncio]

    # Get warnings about missing supplementals
    analysis.warnings
    # => ["Document requires supplemental ':pykka' which is not registered"]

## Available Supplementals

### Python.Pykka

Actor model support via Pykka library

Constructs:

- `:actor_call` - Synchronous actor message (ask pattern)
- `:actor_cast` - Asynchronous actor message (tell pattern)
- `:spawn_actor` - Create new actor instance

Dependencies: `pykka >= 3.0`

Example:

    # Actor call
    {:actor_call, {:variable, "worker"}, "process", [data]}
    # Transforms to: worker.ask({'process': data})

    # Actor cast
    {:actor_cast, {:variable, "worker"}, "log", [message]}
    # Transforms to: worker.tell({'log': message})

    # Spawn actor
    {:spawn_actor, "WorkerActor", [config]}
    # Transforms to: WorkerActor.start(config)

### Python.Asyncio

Async/await patterns via asyncio library

Constructs:

- `:async_await` - Async function execution
- `:async_context` - Async context managers
- `:gather` - Parallel task execution

Dependencies: `asyncio` (stdlib)

Example:

    # Async await
    {:async_operation, :async_await,
      {:function_call, "fetch_data", [url]}}
    # Transforms to: asyncio.run(fetch_data(url))

    # Gather
    {:async_operation, :gather, [
      {:function_call, "fetch_user", [1]},
      {:function_call, "fetch_posts", [1]}
    ]}
    # Transforms to: asyncio.gather(fetch_user(1), fetch_posts(1))

## Creating Supplementals

### Step 1: Implement the Behaviour

    defmodule MyProject.Supplemental.Python.MyLibrary do
      @behaviour Metastatic.Supplemental

      alias Metastatic.Supplemental.Info

      @impl true
      def info do
        %Info{
          name: :my_library,
          language: :python,
          constructs: [:my_construct],
          requires: ["my-library >= 1.0"],
          description: "My library integration"
        }
      end

      @impl true
      def transform(ast, language, opts)

      def transform({:my_construct, args}, :python, _opts) do
        # Transform logic here
        result = {:function_call, "my_library.do_thing", args}
        {:ok, result}
      end

      def transform(_ast, :python, _opts) do
        {:error, {:unsupported_construct, "..."}}
      end

      def transform(_ast, language, _opts) do
        {:error, {:incompatible_language, "..."}}
      end
    end

### Step 2: Write Tests

    defmodule MyProject.Supplemental.Python.MyLibraryTest do
      use ExUnit.Case, async: true

      alias MyProject.Supplemental.Python.MyLibrary

      describe "info/0" do
        test "returns correct metadata" do
          info = MyLibrary.info()
          assert info.name == :my_library
          assert info.language == :python
        end
      end

      describe "transform/3" do
        test "transforms my_construct correctly" do
          ast = {:my_construct, [{:literal, :string, "arg"}]}
          assert {:ok, result} = MyLibrary.transform(ast, :python)
          assert result == {:function_call, "my_library.do_thing",
                            [{:literal, :string, "arg"}]}
        end

        test "returns error for wrong language" do
          ast = {:my_construct, []}
          assert {:error, {:incompatible_language, _}} =
                   MyLibrary.transform(ast, :javascript)
        end
      end
    end

### Step 3: Register

    # Manual registration
    alias Metastatic.Supplemental.Registry

    {:ok, _} = Registry.register(MyProject.Supplemental.Python.MyLibrary)

    # Or via config (auto-registers on startup)
    config :metastatic, :supplementals,
      auto_register: [MyProject.Supplemental.Python.MyLibrary]

### Step 4: Document

Add module documentation with:

- Clear description of what library/pattern it supports
- List of all constructs handled
- Examples showing MetaAST input and output
- Dependency requirements
- Any caveats or limitations

## Best Practices

### Design Guidelines

1. Single Responsibility - One supplemental per library/pattern
2. Explicit Constructs - Use distinct construct atoms (`:actor_call` not `:call`)
3. Language Specific - Target exactly one language per supplemental
4. Idiomatic Output - Generate natural, idiomatic code for target language
5. Comprehensive Tests - Test all constructs, error cases, edge cases

### Naming Conventions

Modules:

    Metastatic.Supplemental.<Language>.<Library>
      ├── Python.Pykka       (library name)
      ├── Python.Asyncio     (pattern name)
      └── JavaScript.RxJS    (library name)

Construct atoms:

    :actor_call          (prefixed by domain)
    :async_await         (verb describing action)
    :spawn_actor         (specific, unambiguous)

Info names:

    name: :pykka         (short, lowercase atom)
    name: :asyncio       (match library name)

### Error Handling

Return specific errors for different failure modes:

    # Unsupported construct
    {:error, {:unsupported_construct, "Pykka does not support: #{inspect(ast)}"}}

    # Wrong language
    {:error, {:incompatible_language, "Pykka only supports Python, got: #{language}"}}

    # Invalid options
    {:error, {:invalid_options, "Unknown option: #{inspect(key)}"}}

### Version Constraints

Use semantic versioning in `requires`:

    requires: [
      "pykka >= 3.0, < 4.0",  # Major version constraint
      "asyncio",               # Stdlib, no version needed
      "aiohttp >= 3.8"         # Minimum version only
    ]

### Testing Strategy

Cover these scenarios:

1. Info validation - Correct metadata structure
2. Happy path - Each construct transforms correctly
3. Error cases - Wrong language, unsupported constructs
4. Edge cases - Empty arguments, nested structures, complex types
5. Options - Handle all option variations
6. Idempotency - Multiple transformations don't break

### Performance Considerations

- Keep transformations fast (<1ms per node)
- Avoid heavy computation in `info/0` (called frequently)
- Use pattern matching for dispatch (faster than conditionals)
- Don't validate AST structure (adapters already do this)

## API Reference

### Behaviour Callbacks

`info/0`

Returns metadata about the supplemental.

    @callback info() :: Info.t()

`transform/3`

Transforms a supplemental construct to target language AST.

    @callback transform(
      ast :: AST.meta_ast(),
      language :: atom(),
      opts :: map()
    ) :: {:ok, AST.meta_ast()} | {:error, term()}

### Registry Functions

`register/1`

Register a supplemental module.

    Registry.register(MySupplemental)
    # => {:ok, :pykka} | {:error, {:already_registered, :pykka}}

`lookup/2`

Find supplemental for construct and language.

    Registry.lookup(:actor_call, :python)
    # => {:ok, Metastatic.Supplemental.Python.Pykka} |
    #    {:error, :not_found}

`all/0`

List all registered supplementals.

    Registry.all()
    # => [Metastatic.Supplemental.Python.Pykka, ...]

### Transformer Functions

`transform/3`

Transform a construct using registered supplementals.

    Transformer.transform(ast, :python)
    # => {:ok, transformed_ast} | {:error, reason}

`available?/2`

Check if a construct is supported for a language.

    Transformer.available?(:actor_call, :python)
    # => true | false

`supported_constructs/1`

Get all supported constructs for a language.

    Transformer.supported_constructs(:python)
    # => [:actor_call, :actor_cast, :spawn_actor, ...]

### Validator Functions

`validate/1`

Analyze a document to detect required supplementals.

    Validator.validate(document)
    # => {:ok, %{required_supplementals: [...], warnings: [...]}}

## Advanced Topics

### Conflict Resolution

When multiple supplementals claim the same construct+language:

    # Registry detects conflicts
    Registry.register(AlternativePykka)
    # => {:error, {:conflict, existing: Pykka, new: AlternativePykka}}

    # Unregister old, register new
    Registry.unregister(Pykka)
    Registry.register(AlternativePykka)

### Chaining Transformations

Supplementals can compose:

    # First supplemental adds intermediate construct
    {:ok, ast1} = Supplemental1.transform(input, :python)

    # Second supplemental further transforms
    {:ok, ast2} = Supplemental2.transform(ast1, :python)

### Stateful Transformations

Pass state via options:

    opts = %{
      actor_prefix: "my_app",
      async_mode: :gather
    }

    Transformer.transform(ast, :python, opts)

### Testing with Fixtures

Create test fixtures for complex scenarios:

    # test/fixtures/python/supplemental/pykka_actors.py
    class Worker(pykka.ThreadingActor):
        def process(self, data):
            return data * 2

    worker = Worker.start()
    result = worker.ask({'process': 42})

## Contributing

Want to add a supplemental?

Use the generator to scaffold a new supplemental:

    mix metastatic.gen.supplemental Python MyLibrary

## Future Work

Planned supplementals:

- Python.Celery - Distributed task queue
- JavaScript.RxJS - Reactive programming
- Ruby.Concurrent - Concurrency primitives
- Go.Channels - CSP-style communication
- Rust.Tokio - Async runtime

← Previous Page Theoretical Foundations
Next Page → Analyzer Plugin System