View Source tflite_beam_litert_compiled_model_isolated (tflite_beam v1.0.0)
A compiled model running on a node of its own.
Everything else here runs LiteRT inside the emulator, which is fast and is the right default. It is also unconditional: a NIF cannot be interrupted, and a segmentation fault inside an accelerator plugin, a delegate that aborts, or an inference that never returns takes the whole virtual machine, every other model in it, and every process that had nothing to do with any of it.
There is no flag that fixes that, because the problem is that the code shares an address space and a scheduler pool with everything else. What does fix it is not sharing them. This module starts a second Erlang node, builds the model there, and forwards calls to it, so a crash costs one node and a supervisor can start another. run/2 and its siblings answer the same way they do on tflite_beam_litert_compiled_model_server, and a dead node comes back as {error, Binary} rather than as a dead caller: this process stays up after its node goes down, answering every call with {error, <<"the isolated models node went down">>}', so that recovering is something the caller decides on rather than something that happens to it. stop/1 when it is done.
What it costs, so the choice is an informed one:
- Inputs and outputs are copied between nodes, twice per call. On a large image that is real, and on a small tensor it is not.
- Starting a node takes hundreds of milliseconds, and the model is built again on it.
- The emulator must be distributed, which means a name and a cookie.
start_link/1starts distribution if it is not already running.
Use it when a model is untrusted, when an accelerator plugin is new, or when one inference must not be able to take the system down. Use the in-process server when the model is yours and the cost is not worth paying.
Summary
Functions
Whether anything is left for the ordinary interpreter to run.
The byte size of each input and output buffer.
The node the model is on.
How many profiling events are waiting, without reading them.
Every profiling event recorded since the last reset.
The most recent Limit events, or all of them when Limit is zero.
Forget the events recorded so far and keep recording.
Run the model over Inputs.
Run the model, with a call timeout.
Run the model and collect whatever counters the accelerator reports.
As run_with_metrics/2, at a given detail level.
As run_with_metrics/3, waiting at most Timeout.
Start one outside a supervision tree.
Start one outside a supervision tree, with gen_server options.
Start a model on a node of its own.
Start a model on a node of its own, with gen_server options.
Stop this process and the node with it.
Per-operator totals, slowest first.
Run a function against the compiled model, on the node that owns it.
As with/2, waiting at most Timeout.
Types
-type opts() :: #{model_path := binary() | string(), runtime_library_dir => binary() | string(), accelerators => [tflite_beam_litert_compiled_model:accelerator()], precision => tflite_beam_litert_compiled_model:precision(), profile => boolean(), signature => tflite_beam_litert_compiled_model:signature_index() | binary() | string(), max_model_bytes => non_neg_integer(), max_queue => non_neg_integer(), peer_args => [string()]}.
Functions
Whether anything is left for the ordinary interpreter to run.
-spec io_sizes(pid()) -> {ok, {[non_neg_integer()], [non_neg_integer()]}} | {error, binary()}.
The byte size of each input and output buffer.
The node the model is on.
For a supervisor that wants to know what it lost, and for a test that wants to kill it.
-spec pending_events(pid()) -> {ok, non_neg_integer()} | {error, binary()}.
How many profiling events are waiting, without reading them.
-spec profile(pid()) -> {ok, [tflite_beam_litert_compiled_model:event()]} | {error, binary()}.
Every profiling event recorded since the last reset.
-spec profile(pid(), tflite_beam_litert_compiled_model:event_limit()) -> {ok, [tflite_beam_litert_compiled_model:event()]} | {error, binary()}.
The most recent Limit events, or all of them when Limit is zero.
Forget the events recorded so far and keep recording.
Run the model over Inputs.
Run the model, with a call timeout.
The timeout ends the wait rather than the inference, the same as it does everywhere else. The difference here is what happens when the node dies under it: the call returns an error and this process stays up, so a caller learns about it instead of being taken down with it.
-spec run_with_metrics(pid(), [binary()]) -> {ok, {[binary()], [{binary(), tflite_beam_litert_compiled_model:metric_value()}]}} | {error, binary()}.
Run the model and collect whatever counters the accelerator reports.
-spec run_with_metrics(pid(), [binary()], tflite_beam_litert_compiled_model:detail_level()) -> {ok, {[binary()], [{binary(), tflite_beam_litert_compiled_model:metric_value()}]}} | {error, binary()}.
As run_with_metrics/2, at a given detail level.
-spec run_with_metrics(pid(), [binary()], tflite_beam_litert_compiled_model:detail_level(), timeout()) -> {ok, {[binary()], [{binary(), tflite_beam_litert_compiled_model:metric_value()}]}} | {error, binary()}.
As run_with_metrics/3, waiting at most Timeout.
Start one outside a supervision tree.
Start one outside a supervision tree, with gen_server options.
Start a model on a node of its own.
Start a model on a node of its own, with gen_server options.
-spec stop(pid()) -> ok.
Stop this process and the node with it.
-spec summarise_profile(pid()) -> {ok, [tflite_beam_litert_compiled_model:summary_entry()]} | {error, binary()}.
Per-operator totals, slowest first.
-spec with(pid(), fun((reference()) -> Result)) -> Result | {error, binary()} when Result :: term().
Run a function against the compiled model, on the node that owns it.
The callback is sent to that node and applied there, so it must return something worth sending back: a value, not a handle to something local to it.
-spec with(pid(), fun((reference()) -> Result), timeout()) -> Result | {error, binary()} when Result :: term().
As with/2, waiting at most Timeout.