View Source Venomous (Venomous v0.1.0)

Venomous is the main module for managing concurrent Python processes using Erlport in an Elixir application.

The core concept revolves around "Snakes" which represent Python worker processes. These Venomous.SnakeWorker are managed and supervised with Venomous.SnakeManager GenServer to allow concurrent and efficient execution of Python code. The Snakes pids and python pids are stored inside :ets table and the Processes are handled by DymanicSupervisor called Venomous.SnakeSupervisor. The unused Snakes get automatically killed by SnakeManager depending on the given configuration.

Main Functionality

  • python/2: The primary function to execute a Python function. It retrieves a Snake (Python worker process) and runs the specified Python function using the arguments provided in a SnakeArgs struct. If no ready Snakes are available, a new one is spawned. If max_children is reached it will return an error with appropriate message.
  • python!/3 | python!/1: Will wait until any Venomous.SnakeWorker is freed, requesting it with the given interval.

Architecture

Venomous consists of several key components:

Configuration Options

The behavior and management of Snakes can be configured through the following options inside :venomous :snake_manager config key:

  • erlport_encoder: %{module: atom(), func: atom(), args: list(any())}: Optional :erlport encoder/decoder python function for converting types.
  • snake_ttl_minutes: non_neg_integer(): Time-to-live for a Snake in minutes. Default is 15 min.
  • perpetual_workers: non_neg_integer(): Number of Snakes to keep alive perpetually. Default is 10.
  • cleaner_interval_ms: non_neg_integer(): Interval in milliseconds for cleaning up inactive Snakes. Default is 60_000 ms.

Auxiliary Functions

Summary

Functions

Clears inactive snakes manually, returns number of snakes cleared

Retrieves x amount of ready snakes. In case of hitting max_children cap, stops and returns all available snakes.

Returns list of :ets table containing alive snakes

Wrapper for python workers Tries to retrieve SnakeWorker which then runs given function inside given module with args. In case of failure will return {:error, message}. In case :EXIT happens, it will kill python worker/process and exit(:normal)

If no Snake is available will continue requesting it with the given interval until any gets freed

Retrieves ready SnakeWorker and python pids. If all processes are busy and exceeds max_children will return {:retrieve_error, message}.

Retrieves ready SnakeWorker and python pids. The worker is then set to :busy until its ran with snake_run(), preventing it from getting removed automatically or used by other process If all processes are busy and exceeds max_children will wait for interval ms and try again.

Kills python process and its SnakeWorker

Runs given %SnakeArgs{} inside given Snake pids.

Functions

@spec clean_inactive_snakes() :: non_neg_integer()

Clears inactive snakes manually, returns number of snakes cleared

Link to this function

get_snakes_ready(amount)

View Source
@spec get_snakes_ready(non_neg_integer()) :: [{pid(), pid()}]

Retrieves x amount of ready snakes. In case of hitting max_children cap, stops and returns all available snakes.

Parameters

  • amount of snakes to retrieve

Returns

  • A list of tuples {pid, pid}
@spec list_alive_snakes() :: [{pid(), pid(), atom(), any()}]

Returns list of :ets table containing alive snakes

Link to this function

python(snake_args, python_timeout \\ 15000)

View Source
@spec python(Venomous.SnakeArgs.t(), non_neg_integer()) :: any()

Wrapper for python workers Tries to retrieve SnakeWorker which then runs given function inside given module with args. In case of failure will return {:error, message}. In case :EXIT happens, it will kill python worker/process and exit(:normal)

Parameters

  • %SnakeArgs{} struct of :module, :func, :args
  • python_timeout \ @default_timeout non_neg_integer() | :infinity Timeout for python call. In case of timeout it will kill python worker/process and return {error: "timeout"}

Returns

  • any() | {error: "timeout"} | {error: any()} retrieves output of python function or error

Link to this function

python!(snake_args, python_timeout \\ 15000, retrieve_interval \\ 200)

View Source

If no Snake is available will continue requesting it with the given interval until any gets freed

@spec retrieve_snake() :: {:retrieve_error, reason :: term()} | {pid(), pid()}

Retrieves ready SnakeWorker and python pids. If all processes are busy and exceeds max_children will return {:retrieve_error, message}.

Returns

  • A tuple {pid, pid} containing the process IDs of the SnakeWorker and python processes. In case of error {:retrieve_error, message}
Link to this function

retrieve_snake!(interval \\ 100)

View Source
@spec retrieve_snake!(non_neg_integer()) :: {pid(), pid()}

Retrieves ready SnakeWorker and python pids. The worker is then set to :busy until its ran with snake_run(), preventing it from getting removed automatically or used by other process If all processes are busy and exceeds max_children will wait for interval ms and try again.

Parameters

  • interval: The time to wait in milliseconds before retrying. Default is @wait_for_snake_interval.

Returns

  • A tuple {pid, pid} containing the process IDs of the SnakeWorker and python processes.
@spec slay_python_worker({pid(), pid()}) :: :ok

Kills python process and its SnakeWorker

Parameters

  • SnakeWorker pid
  • Python pid

Returns

:ok

Link to this function

snake_run(snake_args, arg, python_timeout \\ 15000)

View Source
@spec snake_run(Venomous.SnakeArgs.t(), {pid(), pid()}, non_neg_integer()) :: any()

Runs given %SnakeArgs{} inside given Snake pids.

Parameters

  • %SnakeArgs{} struct of :module, :func, :args
  • {pid, pypid} tuple containing SnakeWorker pid and python pid
  • python_timeout \ @default_timeout non_neg_integer() | :infinity Timeout for python call. In case of timeout it will kill python worker/process and return {error: "timeout"}

Returns

  • any() | {error: "timeout"} | %SnakeError{} retrieves output of python function or error