View Source Venomous (Venomous v0.1.0)
Venomous is the main module for managing concurrent Python processes using Erlport in an Elixir application.
The core concept revolves around "Snakes" which represent Python worker processes. These Venomous.SnakeWorker
are managed and supervised with Venomous.SnakeManager
GenServer to allow concurrent and efficient execution of Python code. The Snakes
pids and python pids are stored inside :ets
table and the Processes are handled by DymanicSupervisor
called Venomous.SnakeSupervisor
. The unused Snakes
get automatically killed by SnakeManager
depending on the given configuration.
Main Functionality
python/2
: The primary function to execute a Python function. It retrieves a Snake (Python worker process) and runs the specified Python function using the arguments provided in aSnakeArgs
struct. If no ready Snakes are available, a new one is spawned. If max_children is reached it will return an error with appropriate message.python!/3
|python!/1
: Will wait until anyVenomous.SnakeWorker
is freed, requesting it with the given interval.
Architecture
Venomous consists of several key components:
Venomous.SnakeWorker
: Manages the execution of Python processes.Venomous.SnakeSupervisor
: A DynamicSupervisor that oversees the SnakeWorkers.Venomous.SnakeManager
: A GenServer that coordinates the SnakeWorkers and handles operations like spawning, retrieval and cleanup.
Configuration Options
The behavior and management of Snakes can be configured through the following options inside :venomous :snake_manager config key:
erlport_encoder: %{module: atom(), func: atom(), args: list(any())}
: Optional :erlport encoder/decoder python function for converting types.snake_ttl_minutes: non_neg_integer()
: Time-to-live for a Snake in minutes. Default is 15 min.perpetual_workers: non_neg_integer()
: Number of Snakes to keep alive perpetually. Default is 10.cleaner_interval_ms: non_neg_integer()
: Interval in milliseconds for cleaning up inactive Snakes. Default is 60_000 ms.
Auxiliary Functions
list_alive_snakes/0
: Returns a list of :ets table containing currently alive Snakes.clean_inactive_snakes/0
: Manually clears inactive Snakes depending on their ttl and returns the number of Snakes cleared.slay_python_worker/1
: Kills a specified Python worker process and its SnakeWorker.
Summary
Functions
Clears inactive snakes manually, returns number of snakes cleared
Retrieves x amount of ready snakes. In case of hitting max_children cap, stops and returns all available snakes.
Returns list of :ets table containing alive snakes
Wrapper for python workers Tries to retrieve SnakeWorker which then runs given function inside given module with args. In case of failure will return {:error, message}. In case :EXIT happens, it will kill python worker/process and exit(:normal)
If no Snake is available will continue requesting it with the given interval until any gets freed
Retrieves ready SnakeWorker and python pids. If all processes are busy and exceeds max_children will return {:retrieve_error, message}.
Retrieves ready SnakeWorker and python pids. The worker is then set to :busy until its ran with snake_run(), preventing it from getting removed automatically or used by other process If all processes are busy and exceeds max_children will wait for interval ms and try again.
Kills python process and its SnakeWorker
Runs given %SnakeArgs{} inside given Snake pids.
Functions
@spec clean_inactive_snakes() :: non_neg_integer()
Clears inactive snakes manually, returns number of snakes cleared
@spec get_snakes_ready(non_neg_integer()) :: [{pid(), pid()}]
Retrieves x amount of ready snakes. In case of hitting max_children cap, stops and returns all available snakes.
Parameters
- amount of snakes to retrieve
Returns
- A list of tuples
{pid, pid}
Returns list of :ets table containing alive snakes
@spec python(Venomous.SnakeArgs.t(), non_neg_integer()) :: any()
Wrapper for python workers Tries to retrieve SnakeWorker which then runs given function inside given module with args. In case of failure will return {:error, message}. In case :EXIT happens, it will kill python worker/process and exit(:normal)
Parameters
- %SnakeArgs{} struct of :module, :func, :args
python_timeout \ @default_timeout non_neg_integer() | :infinity Timeout for python call. In case of timeout it will kill python worker/process and return {error: "timeout"}
Returns
any() | {error: "timeout"} | {error: any()} retrieves output of python function or error
python!(snake_args, python_timeout \\ 15000, retrieve_interval \\ 200)
View Source@spec python!(Venomous.SnakeArgs.t(), non_neg_integer(), non_neg_integer()) :: any()
If no Snake is available will continue requesting it with the given interval until any gets freed
Retrieves ready SnakeWorker and python pids. If all processes are busy and exceeds max_children will return {:retrieve_error, message}.
Returns
- A tuple
{pid, pid}
containing the process IDs of the SnakeWorker and python processes. In case of error{:retrieve_error, message}
@spec retrieve_snake!(non_neg_integer()) :: {pid(), pid()}
Retrieves ready SnakeWorker and python pids. The worker is then set to :busy until its ran with snake_run(), preventing it from getting removed automatically or used by other process If all processes are busy and exceeds max_children will wait for interval ms and try again.
Parameters
- interval: The time to wait in milliseconds before retrying. Default is
@wait_for_snake_interval
.
Returns
- A tuple
{pid, pid}
containing the process IDs of the SnakeWorker and python processes.
Kills python process and its SnakeWorker
Parameters
- SnakeWorker pid
- Python pid
Returns
:ok
@spec snake_run(Venomous.SnakeArgs.t(), {pid(), pid()}, non_neg_integer()) :: any()
Runs given %SnakeArgs{} inside given Snake pids.
Parameters
- %SnakeArgs{} struct of :module, :func, :args
- {pid, pypid} tuple containing SnakeWorker pid and python pid
python_timeout \ @default_timeout non_neg_integer() | :infinity Timeout for python call. In case of timeout it will kill python worker/process and return {error: "timeout"}
Returns
any() | {error: "timeout"} | %SnakeError{} retrieves output of python function or error