View Source Venomous (Venomous v0.2.0)

Venomous is a wrapper around erlport python Ports, designed to simplify concurrent use. It focuses dynamic extensibility, like spawning, reusing and killing processes on demand. Furthermore, unused processes get automatically killed by scheduled process which can be configured inside config.exs. Venomous core functions capture and handle :EXIT calls ensuring that all python process die with it and do not continue their execution.

The core concept revolves around "Snakes" which represent Python worker processes. These Venomous.SnakeWorker are managed and supervised with Venomous.SnakeManager GenServer to allow concurrent and efficient execution of Python code. The Snakes pids and python pids are stored inside :ets table and the Processes are handled by DymanicSupervisor called Venomous.SnakeSupervisor. The unused Snakes get automatically killed by SnakeManager depending on the given configuration.

Main Functionality

  • python/2: The primary function to execute a Python function. It retrieves a Snake (Python worker process) and runs the specified Python function using the arguments provided in a SnakeArgs struct. If no ready Snakes are available, a new one is spawned. If max_children is reached it will return an error with appropriate message.
  • python!/3 | python!/1: Will wait until any Venomous.SnakeWorker is freed, requesting it with the given interval.

Architecture

Venomous consists of several key components:

Configuration Options

The behavior and management of Snakes can be configured through the following options inside :venomous :snake_manager config key:

  • erlport_encoder: %{module: atom(), func: atom(), args: list(any())}: Optional :erlport encoder/decoder python function for converting types. This function is applied to every unnamed python process started by SnakeManager. For more information see Handling Erlport API
  • snake_ttl_minutes: non_neg_integer(): Time-to-live for a Snake in minutes. Default is 15 min.
  • perpetual_workers: non_neg_integer(): Number of Snakes to keep alive perpetually. Default is 10.
  • cleaner_interval: non_neg_integer(): Interval in milliseconds for cleaning up inactive Snakes. Default is 60_000 ms.

Auxiliary Functions

  • list_alive_snakes/0: Returns a list of :ets table containing currently alive Snakes.
  • clean_inactive_snakes/0: Manually clears inactive Snakes depending on their ttl and returns the number of Snakes cleared.
  • slay_python_worker/2: Kills a specified Python worker process and its SnakeWorker. :brutal can be specified as option, which will kill -9 the os process of python which prevents the code from executing until it finalizes or goes through iteration.

Summary

Functions

Clears inactive snakes manually, returns number of snakes cleared.

Retrieves x amount of ready snakes. In case of hitting max_children cap, stops and returns all available snakes.

Returns list of :ets table containing alive snakes

Wrapper for python workers Tries to retrieve Venomous.SnakeWorker which then runs given function inside module with args. In case of failure will return {:error, message}. In case :EXIT happens, it will kill python worker/process and exit(reason)

If no Snake is available will continue requesting it with the given interval until any gets freed or receives :EXIT signal

Retrieves ready SnakeWorker and python pids. If all processes are busy and exceeds max_children will return {:retrieve_error, message}.

Retrieves ready SnakeWorker and python pids. The worker is then set to :busy until its ran with snake_run(), preventing it from getting removed automatically or used by other process If all processes are busy and exceeds max_children will wait for interval ms and try again. Traps the exit signals, to safely escape loop.

Kills python process and its SnakeWorker :brutal also kills the OS process of python, ensuring the process does not continue execution.

Runs given %SnakeArgs{} inside given Snake pids. Takes in a python_timeout which brutally kills the python process after surpassing it. Traps exit and awaits signals [:SNAKE_DONE, :SNAKE_ERROR, :EXIT] In case of an exit, brutally kills the python process ensuring it doesn't get executed any further.

Functions

@spec clean_inactive_snakes() :: non_neg_integer()

Clears inactive snakes manually, returns number of snakes cleared.

Link to this function

get_snakes_ready(amount)

View Source
@spec get_snakes_ready(non_neg_integer()) :: [{pid(), pid()}]

Retrieves x amount of ready snakes. In case of hitting max_children cap, stops and returns all available snakes.

Parameters

  • amount of snakes to retrieve

Returns

  • A list of tuples {pid, pid}
@spec list_alive_snakes() :: [{pid(), pid(), atom(), any()}]

Returns list of :ets table containing alive snakes

Link to this function

python(snake_args, python_timeout \\ 15000)

View Source
@spec python(Venomous.SnakeArgs.t(), non_neg_integer()) :: any()

Wrapper for python workers Tries to retrieve Venomous.SnakeWorker which then runs given function inside module with args. In case of failure will return {:error, message}. In case :EXIT happens, it will kill python worker/process and exit(reason)

Parameters

  • %SnakeArgs{} struct of :module, :func, :args
  • python_timeout \ @default_timeout non_neg_integer() | :infinity Timeout for python call. In case of timeout it will kill python worker/process and return {error: :timeout}

Returns

  • any() | {error: :timeout} | {error: any()} retrieves output of python function or error

Link to this function

python!(snake_args, python_timeout \\ 15000, retrieve_interval \\ 200)

View Source

If no Snake is available will continue requesting it with the given interval until any gets freed or receives :EXIT signal

@spec retrieve_snake() :: {:retrieve_error, reason :: term()} | {pid(), pid()}

Retrieves ready SnakeWorker and python pids. If all processes are busy and exceeds max_children will return {:retrieve_error, message}.

Returns

  • A tuple {pid, pid} containing the process IDs of the SnakeWorker and python processes. In case of error {:retrieve_error, message}
Link to this function

retrieve_snake!(interval \\ 100)

View Source
@spec retrieve_snake!(non_neg_integer()) :: {pid(), pid()}

Retrieves ready SnakeWorker and python pids. The worker is then set to :busy until its ran with snake_run(), preventing it from getting removed automatically or used by other process If all processes are busy and exceeds max_children will wait for interval ms and try again. Traps the exit signals, to safely escape loop.

Parameters

  • interval: The time to wait in milliseconds before retrying. Default is @wait_for_snake_interval.

Returns

  • A tuple {pid, pid} containing the process IDs of the SnakeWorker and python processes.
Link to this function

slay_python_worker(arg, termination_style \\ :peaceful)

View Source
@spec slay_python_worker(
  {pid(), pid()},
  atom() | {pid(), pid()}
) :: :ok

Kills python process and its SnakeWorker :brutal also kills the OS process of python, ensuring the process does not continue execution.

Parameters

  • SnakeWorker pid, Python pid tuple
  • a Way to kill process. default: :brutal additionally kills with kill -9 ensuring the python does not execute further

Returns

:ok

Link to this function

snake_run(snake_args, arg, python_timeout \\ 15000)

View Source
@spec snake_run(Venomous.SnakeArgs.t(), {pid(), pid()}, non_neg_integer()) :: any()

Runs given %SnakeArgs{} inside given Snake pids. Takes in a python_timeout which brutally kills the python process after surpassing it. Traps exit and awaits signals [:SNAKE_DONE, :SNAKE_ERROR, :EXIT] In case of an exit, brutally kills the python process ensuring it doesn't get executed any further.

Parameters

  • %SnakeArgs{} struct of :module, :func, :args
  • {pid, pypid} tuple containing SnakeWorker pid and python pid
  • python_timeout \ @default_timeout non_neg_integer() | :infinity Timeout for python call. In case of timeout it will kill python worker/process and return {error: :timeout}

Returns

  • any() | {error: :timeout} | %SnakeError{} retrieves output of python function or error