View Source tflite_beam_litert_compiled_model_server (tflite_beam v1.0.0)
A compiled model that lives inside a process, so that no two callers share its buffers.
tflite_beam_litert_compiled_model is not wrong, and it mirrors LiteRT's C API faithfully. What it does not say anywhere is that a compiled model owns one set of input and output buffers for its whole life, allocated when it is built rather than per call. run/2 writes the caller's input into those buffers, runs, and reads the outputs back out of them.
LiteRT states that its compiled model API is not verified for multithreading, and the profile buffer underneath says outright that it is not thread safe, so two callers inside one model at once is a data race and not merely crossed outputs. The direct module therefore refuses a second concurrent caller rather than admitting it, and before it did, four processes running twenty-five inferences each against one shared model got a handful of answers belonging to a different process, with nothing to say which ones.
Refusal is honest but it is not a queue. This module is the queue: callers wait their turn instead of being told to come back.
This is a different hazard from the one tflite_beam_interpreter_server answers. There the danger is a three-step sequence being interleaved; here run/2 is a single call and interleaving still loses, because the state that races is the buffers behind it. Holding the model in one process is the same answer to both.
The profile is per model, not per call, so summarise_profile/1 here reports the runs since the last reset_profile/1 rather than the last one. That is usually what you want from a server: a shape over many calls rather than one sample. The buffer under it is fixed at 512 * 1024 entries, so it is bounded but not tight. Reset when you want to measure a change.
Summary
Functions
Whether one accelerator claimed the whole graph.
The byte size of each input and output buffer.
How many profiling events are waiting, without reading them.
Every profiling event recorded since the last reset.
The most recent Limit profiling events, or all of them when Limit is zero.
Forget the events recorded so far and keep recording.
Run the model over Inputs and return its outputs.
Run the model, with a call timeout.
Run the model and collect whatever counters the accelerator reports.
Run the model with metrics collection bracketing the inference.
Run the model with metrics, with a call timeout.
Start a compiled model process outside a supervision tree.
Start a compiled model process outside a supervision tree.
Start a compiled model process, on the CPU with no profiling.
Start a compiled model process.
Stop the process, and with it the compiled model.
Per-operator totals over every run since the last reset, slowest first.
Run a function against the compiled model inside the owning process.
Run a function against the compiled model inside the owning process.
Types
-type opts() :: #{accelerators => [tflite_beam_litert_compiled_model:accelerator()], precision => tflite_beam_litert_compiled_model:precision(), profile => boolean(), signature => tflite_beam_litert_compiled_model:signature_index() | binary() | string(), max_model_bytes => non_neg_integer(), max_queue => non_neg_integer()}.
Functions
Whether one accelerator claimed the whole graph.
-spec io_sizes(pid()) -> {ok, {[non_neg_integer()], [non_neg_integer()]}} | {error, binary()}.
The byte size of each input and output buffer.
-spec pending_events(pid()) -> {ok, non_neg_integer()} | {error, binary()}.
How many profiling events are waiting, without reading them.
-spec profile(pid()) -> {ok, [tflite_beam_litert_compiled_model:event()]} | {error, binary()}.
Every profiling event recorded since the last reset.
-spec profile(pid(), tflite_beam_litert_compiled_model:event_limit()) -> {ok, [tflite_beam_litert_compiled_model:event()]} | {error, binary()}.
The most recent Limit profiling events, or all of them when Limit is zero.
A server that runs for a long time is exactly the case where the bound is worth using: nothing trims the profile except reset_profile/1. The guard is here as well as in the direct module for the same reason it is on run_with_metrics/4: an argument the direct module refuses with function_clause would raise inside this server and take the model with it.
Forget the events recorded so far and keep recording.
Run the model over Inputs and return its outputs.
Concurrent callers are serialised by the process rather than racing over the model's buffers, so each gets the answer to its own input.
Run the model, with a call timeout.
The timeout gives up on the answer; it does not stop the work. When it runs out this exits the calling process, which is gen_server:call/3 behaviour, and the server carries on with the inference it was given. Anything queued behind it still waits. Raise the timeout rather than retry: a retry joins the queue behind the call it replaced.
-spec run_with_metrics(pid(), [binary()]) -> {ok, {[binary()], [{binary(), term()}]}} | {error, binary()}.
Run the model and collect whatever counters the accelerator reports.
-spec run_with_metrics(pid(), [binary()], tflite_beam_litert_compiled_model:detail_level()) -> {ok, {[binary()], [{binary(), term()}]}} | {error, binary()}.
Run the model with metrics collection bracketing the inference.
-spec run_with_metrics(pid(), [binary()], tflite_beam_litert_compiled_model:detail_level(), timeout()) -> {ok, {[binary()], [{binary(), term()}]}} | {error, binary()}.
Run the model with metrics, with a call timeout.
The guard is here as well as in the direct module on purpose: an argument the direct module refuses with function_clause would raise inside this server's handle_call and take the server down with it, so a caller mistake would cost the model rather than the call.
Start a compiled model process outside a supervision tree.
Start a compiled model process outside a supervision tree.
Start a compiled model process, on the CPU with no profiling.
Start a compiled model process.
Opts is what tflite_beam_litert_compiled_model:new/3 takes. The environment is created by the caller and may be shared between servers: it carries where accelerator plugins are found and nothing per-model.
-spec stop(pid()) -> ok.
Stop the process, and with it the compiled model.
-spec summarise_profile(pid()) -> {ok, [tflite_beam_litert_compiled_model:summary_entry()]} | {error, binary()}.
Per-operator totals over every run since the last reset, slowest first.
Run a function against the compiled model inside the owning process.
For the sequences run/2 does not cover, such as resetting the profile and running a measured batch as one uninterrupted step.
What this guarantees is narrow and worth stating exactly: the function runs in this process, so the server handles no other message while it does. A callback that raises is caught and returned as an error, because losing a compiled model to somebody else's mistake is not a reasonable price. It is not a sandbox beyond that. A callback that keeps the reference and hands the model to another process with tflite_beam_litert_compiled_model:controlling_process/2 leaves this server alive and unable to use its own model. And a timeout on with/3 ends the wait, not the callback, which carries on holding the server.
Run a function against the compiled model inside the owning process.