Snex (Snex v0.1.0)
View SourceEasy and efficient Python interop for Elixir.
Highlights
- 🛡️ Robust & Isolated: Run multiple Python interpreters in separate OS processes, preventing GIL issues from affecting your Elixir application.
- 📦 Declarative Environments:
Leverages
uv
to manage Python versions and dependencies, embedding them into your application's release for consistent deployments. - ✨ Ergonomic Interface: A powerful and efficient interface with explicit control over data passing between Elixir and Python processes.
- 🤸 Flexible:
Supports custom Python environments,
asyncio
code, and integration with external Python projects. - ⏩ Forward Compatibility: Built on stable foundations, so future versions of Python or Elixir are unlikely to require Snex updates to use — they should work day one!
Quick example
defmodule SnexTest.NumpyInterpreter do
use Snex.Interpreter,
pyproject_toml: """
[project]
name = "my-numpy-project"
version = "0.0.0"
requires-python = "==3.11.*"
dependencies = ["numpy>=2"]
"""
end
iex> {:ok, inp} = SnexTest.NumpyInterpreter.start_link()
iex> {:ok, env} = Snex.make_env(inp)
...>
iex> Snex.pyeval(env, """
...> import numpy as np
...> matrix = np.fromfunction(lambda i, j: (-1) ** (i + j), (s, s), dtype=int)
...> """, %{"s" => 6}, returning: "np.linalg.norm(matrix)")
{:ok, 6.0}
Installation & Requirements
- Elixir
>= 1.18
- uv - a fast Python package & project manager, used by Snex to create and manage Python environments. It has to be available at compilation time but isn't needed at runtime.
- Python
>= 3.11
- this is the minimum supported version you can run your scripts with. You don't need to have it installed — Snex will fetch it withuv
.
def deps do
[
{:snex, "~> 0.1.0"}
]
end
Core Concepts & Usage
Custom Interpreter
You can define your Python project settings using use Snex.Interpreter
in your module.
Set a required Python version and any dependencies —both the Python binary & the dependencies will be fetched & synced at compile time with uv, and put into your application's priv directory.
defmodule SnexTest.NumpyInterpreter do
use Snex.Interpreter,
pyproject_toml: """
[project]
name = "my-numpy-project"
version = "0.0.0"
requires-python = "==3.11.*"
dependencies = ["numpy>=2"]
"""
end
The modules using Snex.Interpreter
have to be start_link
ed to use.
Each Snex.Interpreter
(BEAM) process manages a separate Python (OS) process.
Snex.pyeval
The main way of interacting with the interpreter process is Snex.pyeval/4
(and other arities).
This is the function that runs Python code, returns data from the interpreter, and more.
iex> {:ok, interpreter} = SnexTest.NumpyInterpreter.start_link()
iex> {:ok, %Snex.Env{} = env} = Snex.make_env(interpreter)
...>
iex> Snex.pyeval(env, """
...> import numpy as np
...> matrix = np.fromfunction(lambda i, j: (-1) ** (i + j), (6, 6), dtype=int)
...> scalar = np.linalg.norm(matrix)
...> """, returning: "scalar")
{:ok, 6.0}
The :returning
option can take any valid Python expression, or an Elixir list of them:
iex> {:ok, interpreter} = SnexTest.NumpyInterpreter.start_link()
iex> {:ok, env} = Snex.make_env(interpreter)
...>
iex> Snex.pyeval(env, "x = 3", returning: ["x", "x*2", "x**2"])
{:ok, [3, 6, 9]}
Environments
Snex.Env
struct, also called "environment", is an Elixir-side reference to Python-side variable context in which your Python code will run.
New environments can be allocated with Snex.make_env/3
(and other arities).
Environments are mutable, and will be modified by your Python code. In Python parlance, they are global & local symbol table your Python code is executed with.
Important
Environments are garbage collected
When a %Snex.Env{}
value is cleaned up by the BEAM VM, the Python process is signalled to deallocate the environment associated with that value.
Reusing a single environment, you can use variables defined in the previous Snex.pyeval/4
calls:
iex> {:ok, inp} = SnexTest.NumpyInterpreter.start_link()
iex> {:ok, env} = Snex.make_env(inp)
...>
...> # `pyeval` does not return a value if not given a `returning` opt
iex> :ok = Snex.pyeval(env, "x = 10")
...>
...> # additional data can be provided for `pyeval` to put in the environment
...> # before running the code
iex> :ok = Snex.pyeval(env, "y = x * z", %{"z" => 2})
...>
...> # `pyeval` can also be called with `:returning` opt alone
iex> Snex.pyeval(env, returning: ["x", "y", "z"])
{:ok, [10, 20, 2]}
Using Snex.make_env/2
and Snex.make_env/3
, you can also create a new environment:
- copying variables from an old environment
Snex.make_env(interpreter, from: old_env)
- copying variables from multiple environments (later override previous)
Snex.make_env(interpreter, from: [ oldest_env, {older_env, only: ["pool"]}, {old_env, except: ["pool"]} ]))
- setting some initial variables (taking precedence over variables from
:from
)Snex.make_env(interpreter, %{"hello" => 42.0}, from: {old_env, only: ["world"]})
Warning
The environments you copy from have to belong to the same interpreter!
JSON serialization
User data sent between Python and Elixir is subject to "standard" JSON serialization and deserialization using JSON
on the Elixir side and json
on the Python side.
Among other things, this means that Python tuples will be serialized as arrays, while Elixir atoms and binaries will be serialized as strings.
iex> {:ok, inp} = SnexTest.NumpyInterpreter.start_link()
iex> {:ok, env} = Snex.make_env(inp)
...>
iex> Snex.pyeval(env, "x = ('hello', y)", %{"y" => :world}, returning: "x")
{:ok, ["hello", "world"]}
Run async code
Code ran by Snex lives in an asyncio
loop.
You can include async functions in your snippets and await them on the top level:
iex> {:ok, inp} = SnexTest.NumpyInterpreter.start_link()
iex> {:ok, env} = Snex.make_env(inp)
...>
iex> Snex.pyeval(env, """
...> import asyncio
...> async def do_thing():
...> await asyncio.sleep(0.01)
...> return "hello"
...>
...> result = await do_thing()
...> """, returning: ["result"])
{:ok, "hello"}
Run blocking code
A good way to run any blocking code is to prepare and use your own thread or process pools:
iex> {:ok, inp} = SnexTest.NumpyInterpreter.start_link()
iex> {:ok, pool_env} = Snex.make_env(inp)
...>
iex> :ok = Snex.pyeval(pool_env, """
...> import asyncio
...> from concurrent.futures import ThreadPoolExecutor
...>
...> pool = ThreadPoolExecutor(max_workers=cnt)
...> loop = asyncio.get_running_loop()
...> """, %{"cnt" => 5})
...>
...> # You can keep the pool environment around and copy it into new ones
iex> {:ok, env} = Snex.make_env(inp, from: {pool_env, only: ["pool", "loop"]})
...>
iex> {:ok, "world!"} = Snex.pyeval(env, """
...> def blocking_io():
...> return "world!"
...>
...> res = await loop.run_in_executor(pool, blocking_io)
...> """, returning: "res")
{:ok, "world!"}
Use your in-repo project
You can reference your existing project path in use Snex.Interpreter
.
The existing pyproject.toml
and uv.lock
will be used to seed the Python environment.
defmodule SnexTest.MyProject do
use Snex.Interpreter,
project_path: "test/my_python_proj"
end
# $ cat test/my_python_proj/foo.py
# def bar():
# return "hi from bar"
# Provide the project's path at runtime - you'll likely want to use
# :code.priv_dir(:your_otp_app) and construct a path relative to that.
iex> {:ok, inp} = SnexTest.MyProject.start_link(environment: %{
...> "PYTHONPATH" => "test/my_python_proj"
...> })
iex> {:ok, env} = Snex.make_env(inp)
...>
iex> Snex.pyeval(env, "import foo", returning: "foo.bar()")
{:ok, "hi from bar"}
Summary
Types
A map of additional variables to be added to the environment.
A string of Python code to be evaluated.
An "environment" is an Elixir-side reference to Python-side variable context in which your Python code will run.
A single environment or a list of environments to copy variables from.
Option for Snex.make_env/3
.
Option for Snex.pyeval/4
.
Functions
Shorthand for Snex.make_env/3
Shorthand for Snex.make_env/3
Creates a new environment, %Snex.Env{}
.
Shorthand for Snex.pyeval/4
Shorthand for Snex.pyeval/4
Evaluates a Python code string in the given environment.
Types
A map of additional variables to be added to the environment.
See Snex.make_env/3
.
@type code() :: String.t()
A string of Python code to be evaluated.
See Snex.pyeval/4
.
@opaque env()
An "environment" is an Elixir-side reference to Python-side variable context in which your Python code will run.
See Snex.make_env/3
for more information.
A single environment or a list of environments to copy variables from.
See Snex.make_env/3
.
Option for Snex.make_env/3
.
Option for Snex.pyeval/4
.
Functions
@spec make_env(Snex.Interpreter.server()) :: {:ok, env()} | {:error, Snex.Error.t() | any()}
Shorthand for Snex.make_env/3
:
Snex.make_env(interpreter, %{} = _additional_vars, [] = _opts)
@spec make_env( Snex.Interpreter.server(), additional_vars() | [make_env_opt()] ) :: {:ok, env()} | {:error, Snex.Error.t() | any()}
Shorthand for Snex.make_env/3
:
# when given a map of `additional_vars`:
Snex.make_env(interpreter, additional_vars, [] = _opts)
# when given an `opts` list:
Snex.make_env(interpreter, %{} = _additional_vars, opts)
@spec make_env( Snex.Interpreter.server(), additional_vars(), [make_env_opt()] ) :: {:ok, env()} | {:error, Snex.Error.t() | any()}
Creates a new environment, %Snex.Env{}
.
A %Snex.Env{}
instance is an Elixir-side reference to a variable context in Python.
The variable contexts are the global & local symbol table the Python code will be executed
with using the Snex.pyeval/2
function.
additional_vars
are additional variables that will be added to the environment.
They will be applied after copying variables from the environments listed in the :from
option.
Returns a tuple {:ok, %Snex.Env{}}
on success.
Options
:from
- a list of environments to copy variables from. Each value in the list can be either a tuple{%Snex.Env{}, opts}
, or a%Snex.Env{}
(Shorthand for{%Snex.Env{}, []}
).The following mutually exclusive options are supported:
:only
- a list of variable names to copy from thefrom
environment.:except
- a list of variable names to exclude from thefrom
environment.
Examples
# Create a new empty environment
Snex.make_env(interpreter)
# Create a new environment with additional variables
Snex.make_env(interpreter, %{"x" => 1, "y" => 2})
# Create a new environment copying variables from existing environments
Snex.make_env(interpreter, from: env)
Snex.make_env(interpreter, from: {env, except: ["y"]})
Snex.make_env(interpreter, from: [env1, {env2, only: ["x"]}])
# Create a new environment with both additional variables and `:from`
Snex.make_env(interpreter, %{"x" => 1, "y" => 2}, from: env)
@spec pyeval( env(), code() | additional_vars() | [pyeval_opt()] ) :: :ok | {:ok, any()} | {:error, Snex.Error.t() | any()}
Shorthand for Snex.pyeval/4
:
# when given a `code` string:
Snex.pyeval(env, code, %{} = _additional_vars, [] = _opts)
# when given an `additional_vars` map:
Snex.pyeval(env, nil = _code, additional_vars, [] = _opts)
# when given an `opts` list:
Snex.pyeval(env, nil = _code, %{} = _additional_vars, opts)
@spec pyeval( env(), code() | nil, additional_vars() | [pyeval_opt()] ) :: :ok | {:ok, any()} | {:error, Snex.Error.t() | any()}
Shorthand for Snex.pyeval/4
:
# when given an `additional_vars` map:
Snex.pyeval(env, code, additional_vars, [] = _opts)
# when given an `opts` list:
Snex.pyeval(env, code, %{} = _additional_vars, opts)
@spec pyeval( env(), code() | nil, additional_vars(), [pyeval_opt()] ) :: :ok | {:ok, any()} | {:error, Snex.Error.t() | any()}
Evaluates a Python code string in the given environment.
additional_vars
are added to the environment before the code is executed.
See Snex.make_env/3
for more information.
Returns :ok
on success, or a tuple {:ok, result}
if :returning
option is provided.
Options
:returning
- a Python expression or a list of Python expressions to evaluate and return from this function. If not provided, the result will be:ok
.:timeout
- the timeout for the evaluation. Can be atimeout()
or:infinity
. Default: 5 seconds.
Examples
Snex.pyeval(env, """
res = [x for x in range(num_range)]
""", %{"num_range" => 6}, returning: "[x * x for x in res]")
[0, 1, 4, 9, 16, 25]