View Source TFLiteElixir.LiteRT.CompiledModel.Server (tflite_elixir v1.0.0-rc4)
A compiled model that lives inside a process, so several processes can share one model without taking turns badly.
TFLiteElixir.LiteRT.CompiledModel refuses a second concurrent caller, which
is safe but leaves the caller to arrange the turns. This does the arranging:
calls are serialised by the process, and the model is claimed by it, so a
reference that escaped cannot be used behind its back.
{:ok, env} = TFLiteElixir.LiteRT.CompiledModel.environment()
{:ok, server} = TFLiteElixir.LiteRT.CompiledModel.Server.start_link(env, path)
{:ok, outputs} = TFLiteElixir.LiteRT.CompiledModel.Server.run(server, inputs)A model still runs one inference at a time, because LiteRT does; what the process adds is that waiting is explicit and bounded rather than a race.
The queue is bounded
A caller that submits faster than the model runs would otherwise grow the
mailbox until the node dies. Past :max_queue pending calls the server
answers {:error, "the model's queue is full"} instead, which is a back
pressure signal a caller can act on. The default is 64.
Summary
Functions
Whether the accelerator took the whole graph.
Raising version of fully_accelerated/1.
As fully_accelerated/1, answering false rather than an error.
The byte size of each input and output tensor, as {inputs, outputs}.
Raising version of io_sizes/1.
How many profiling events are waiting, without reading them.
Raising version of pending_events/1.
Every profiling event recorded so far.
The most recent limit profiling events, or all of them when zero.
Raising version of profile/1.
Raising version of profile/2.
Forget the events recorded so far and keep recording.
Raising version of reset_profile/1.
Run the model over a list of input binaries.
Run the model, waiting at most timeout.
Raising version of run/2.
Raising version of run/3.
Run and collect whatever counters the accelerator reports.
As run_with_metrics/2, at a given detail level.
As run_with_metrics/3, waiting at most timeout.
Raising version of run_with_metrics/2.
Raising version of run_with_metrics/3.
Start one outside a supervision tree.
Start one outside a supervision tree, with options.
Start a compiled model process linked to the caller.
Start a compiled model process linked to the caller.
Stop the process, and with it the compiled model.
Per-operator totals over every run since the last reset, slowest first.
Raising version of summarise_profile/1.
Run a function against the compiled model inside the owning process.
As with/2, waiting at most timeout.
Types
@type opts() :: [{:max_queue, non_neg_integer()} | {atom(), term()}]
Functions
Whether the accelerator took the whole graph.
Raising version of fully_accelerated/1.
As fully_accelerated/1, answering false rather than an error.
@spec io_sizes(pid()) :: {:ok, {[non_neg_integer()], [non_neg_integer()]}} | {:error, String.t()}
The byte size of each input and output tensor, as {inputs, outputs}.
Raising version of io_sizes/1.
@spec pending_events(pid()) :: {:ok, non_neg_integer()} | {:error, String.t()}
How many profiling events are waiting, without reading them.
Raising version of pending_events/1.
@spec profile(pid()) :: {:ok, [TFLiteElixir.LiteRT.CompiledModel.event()]} | {:error, String.t()}
Every profiling event recorded so far.
The profile belongs to the model, not to a call, so this covers every run any process has made since the last reset.
@spec profile(pid(), non_neg_integer()) :: {:ok, [TFLiteElixir.LiteRT.CompiledModel.event()]} | {:error, String.t()}
The most recent limit profiling events, or all of them when zero.
Raising version of profile/1.
Raising version of profile/2.
Forget the events recorded so far and keep recording.
Raising version of reset_profile/1.
Run the model over a list of input binaries.
Run the model, waiting at most timeout.
Raising version of run/2.
Raising version of run/3.
@spec run_with_metrics(pid(), [binary()]) :: {:ok, {[binary()], [{binary(), TFLiteElixir.LiteRT.CompiledModel.metric_value()}]}} | {:error, String.t()}
Run and collect whatever counters the accelerator reports.
@spec run_with_metrics(pid(), [binary()], non_neg_integer()) :: {:ok, {[binary()], [{binary(), TFLiteElixir.LiteRT.CompiledModel.metric_value()}]}} | {:error, String.t()}
As run_with_metrics/2, at a given detail level.
@spec run_with_metrics(pid(), [binary()], non_neg_integer(), timeout()) :: {:ok, {[binary()], [{binary(), TFLiteElixir.LiteRT.CompiledModel.metric_value()}]}} | {:error, String.t()}
As run_with_metrics/3, waiting at most timeout.
Raising version of run_with_metrics/2.
Raising version of run_with_metrics/3.
Start one outside a supervision tree.
Start one outside a supervision tree, with options.
Start a compiled model process linked to the caller.
Start a compiled model process linked to the caller.
Takes everything TFLiteElixir.LiteRT.CompiledModel.new/3 takes, plus:
:max_queue. How many calls may be waiting before further ones are refused. Defaults to 64.
@spec stop(pid()) :: :ok
Stop the process, and with it the compiled model.
@spec summarise_profile(pid()) :: {:ok, [TFLiteElixir.LiteRT.CompiledModel.summary_entry()]} | {:error, String.t()}
Per-operator totals over every run since the last reset, slowest first.
Raising version of summarise_profile/1.
Run a function against the compiled model inside the owning process.
The escape hatch for anything this module does not forward. The reference is only usable for the duration of the call, because the server owns the model and takes it back afterwards. A function that raises costs the call and not the model.
@spec with(pid(), (reference() -> result), timeout()) :: result | {:error, String.t()} when result: term()
As with/2, waiting at most timeout.