A compiled model that lives inside a process, so several processes can share one model without taking turns badly.
TFLiteElixir.LiteRT.CompiledModel refuses a second concurrent caller, which
is safe but leaves the caller to arrange the turns. This does the arranging:
calls are serialised by the process, and the model is claimed by it, so a
reference that escaped cannot be used behind its back.
{:ok, env} = TFLiteElixir.LiteRT.CompiledModel.environment()
{:ok, server} = TFLiteElixir.LiteRT.CompiledModel.Server.start_link(env, path)
{:ok, outputs} = TFLiteElixir.LiteRT.CompiledModel.Server.run(server, inputs)A model still runs one inference at a time, because LiteRT does; what the process adds is that waiting is explicit and bounded rather than a race.
The queue is bounded
A caller that submits faster than the model runs would otherwise grow the
mailbox until the node dies. Past :max_queue pending calls the server
answers {:error, "the model's queue is full"} instead, which is a back
pressure signal a caller can act on. The default is 64.
Summary
Functions
Whether every operation was delegated, including to CPU delegates such as XNNPACK.
Raising version of fully_accelerated/1.
As fully_accelerated/1, answering false rather than an error.
The byte size of each input and output tensor, as {inputs, outputs}.
Raising version of io_sizes/1.
Whether every operation executes on GPU or NPU, with no CPU execution.
Raising version of non_cpu_fully_accelerated/1.
As non_cpu_fully_accelerated/1, returning false if the query fails.
How many profiling events are waiting, without reading them.
Raising version of pending_events/1.
Every profiling event recorded so far.
The most recent limit profiling events, or all of them when zero.
Raising version of profile/1.
Raising version of profile/2.
Forget the events recorded so far and keep recording.
Raising version of reset_profile/1.
Run the model over a list of input binaries.
Run the model, waiting at most timeout.
Raising version of run/2.
Raising version of run/3.
Run and collect whatever counters the accelerator reports.
As CompiledModel.run_with_metrics/3, with an integer detail level or metrics options.
As run_with_metrics/3, waiting at most timeout.
Raising version of run_with_metrics/2.
Raising version of run_with_metrics/3.
Raising version of run_with_metrics/4.
Start one outside a supervision tree.
Start one outside a supervision tree, with options.
Start a compiled model process linked to the caller.
Start a compiled model process linked to the caller.
Stop the process, and with it the compiled model.
Per-operator totals over every run since the last reset, slowest first.
Raising version of summarise_profile/1.
Run a function against the compiled model inside the owning process.
As with/2, waiting at most timeout.
Types
@type opts() :: [{:max_queue, non_neg_integer()} | {atom(), term()}] | map()
Functions
Whether every operation was delegated, including to CPU delegates such as XNNPACK.
Raising version of fully_accelerated/1.
As fully_accelerated/1, answering false rather than an error.
@spec io_sizes(pid()) :: {:ok, {[non_neg_integer()], [non_neg_integer()]}} | {:error, String.t()}
The byte size of each input and output tensor, as {inputs, outputs}.
Raising version of io_sizes/1.
Whether every operation executes on GPU or NPU, with no CPU execution.
Raising version of non_cpu_fully_accelerated/1.
As non_cpu_fully_accelerated/1, returning false if the query fails.
@spec pending_events(pid()) :: {:ok, non_neg_integer()} | {:error, String.t()}
How many profiling events are waiting, without reading them.
Raising version of pending_events/1.
@spec profile(pid()) :: {:ok, [TFLiteElixir.LiteRT.CompiledModel.event()]} | {:error, String.t()}
Every profiling event recorded so far.
The profile belongs to the model, not to a call, so this covers every run any process has made since the last reset.
@spec profile(pid(), non_neg_integer()) :: {:ok, [TFLiteElixir.LiteRT.CompiledModel.event()]} | {:error, String.t()}
The most recent limit profiling events, or all of them when zero.
Raising version of profile/1.
Raising version of profile/2.
Forget the events recorded so far and keep recording.
Raising version of reset_profile/1.
Run the model over a list of input binaries.
Run the model, waiting at most timeout.
Raising version of run/2.
Raising version of run/3.
@spec run_with_metrics(pid(), [binary()]) :: {:ok, {[binary()], [{binary(), TFLiteElixir.LiteRT.CompiledModel.metric_value()}]}} | {:error, String.t()}
Run and collect whatever counters the accelerator reports.
@spec run_with_metrics( pid(), [binary()], TFLiteElixir.LiteRT.CompiledModel.detail_level() | TFLiteElixir.LiteRT.CompiledModel.metrics_options() ) :: {:ok, {[binary()], [{binary(), TFLiteElixir.LiteRT.CompiledModel.metric_value()}]}} | {:error, String.t()}
As CompiledModel.run_with_metrics/3, with an integer detail level or metrics options.
@spec run_with_metrics( pid(), [binary()], TFLiteElixir.LiteRT.CompiledModel.detail_level() | TFLiteElixir.LiteRT.CompiledModel.metrics_options(), timeout() ) :: {:ok, {[binary()], [{binary(), TFLiteElixir.LiteRT.CompiledModel.metric_value()}]}} | {:error, String.t()}
As run_with_metrics/3, waiting at most timeout.
Raising version of run_with_metrics/2.
Raising version of run_with_metrics/3.
Raising version of run_with_metrics/4.
Start one outside a supervision tree.
Start one outside a supervision tree, with options.
Start a compiled model process linked to the caller.
Start a compiled model process linked to the caller.
Takes everything TFLiteElixir.LiteRT.CompiledModel.new/3 takes, plus:
:max_queue. How many calls may be waiting before further ones are refused. Defaults to 64.
@spec stop(pid()) :: :ok
Stop the process, and with it the compiled model.
@spec summarise_profile(pid()) :: {:ok, [TFLiteElixir.LiteRT.CompiledModel.summary_entry()]} | {:error, String.t()}
Per-operator totals over every run since the last reset, slowest first.
Raising version of summarise_profile/1.
Run a function against the compiled model inside the owning process.
The callback runs in the server process and must return promptly. A function
that raises costs the call, not the model. A timeout ends the caller's wait;
it does not cancel the callback. The callback can transfer ownership with
CompiledModel.controlling_process/2, leaving the server unable to use its
model, so with/2 is not an ownership sandbox.
@spec with(pid(), (reference() -> result), timeout()) :: result | {:error, String.t()} when result: term()
As with/2, waiting at most timeout.