View Source tflite_beam_litert_compiled_model (tflite_beam v1.0.0)
LiteRT's compiled model, as it is.
This is not the faster way to run a model. Measured on one machine against mobilenet_v2_1.0_224, a compiled model on the GPU and an interpreter with the same plugin attached through tflite_beam_delegate:external/2 land in the same band, because underneath they are the same delegate. Reach for this when you want to see **where the time went**: a profiler can be attached to a compiled model and to nothing else, and it reports every operator, how long it took, and whether an accelerator or the CPU ran it.
A model owns its input and output buffers, allocated once when it is built, so run/2 writes into those buffers, runs, and reads back out of them. LiteRT does not promise its compiled model API is safe to enter from two threads, so a second caller arriving while one is inside the model is **refused** with {error, <<"compiled model is in use by another caller">>} rather than admitted. That is a refusal, not a queue: callers who would rather wait their turn want tflite_beam_litert_compiled_model_server.
Summary
Types
How many recent events profile/2 answers with, zero meaning all of them.
What a profiling event is. An integer means this build met a type LiteRT has since added, which is reported rather than dropped.
The signature index crosses into the NIF through enif_get_int, so the range is the C int range and not every non_neg_integer().
Same range and for the same reason: it crosses into the NIF as a C int.
Whatever the accelerator chose to report. unsupported is a value of a kind LiteRtAny can hold but this library has no term for.
Functions
Whether this build has the LiteRT API at all.
Which process this model belongs to, or undefined if nobody has claimed it. Answers {error, _} while another caller is inside the model, because the answer is read under the same lock that call holds.
Give the model to Pid, after which every other process is refused.
An environment with no directory to look for accelerator plugins in.
An environment that looks for accelerator plugins in Dir.
Whether anything is left for the ordinary interpreter to run.
The byte size of each input and output buffer.
A compiled model on the CPU, with no profiling.
Compile Path in Env.
How many profiling events are waiting, without reading them.
What this build of the library can reach.
Every profiling event recorded so far, oldest first.
The most recent Limit profiling events, oldest first, or all of them when Limit is zero.
Forget the events recorded so far and keep recording.
Run the model over Inputs and return its outputs.
Run the model and collect whatever counters the accelerator reports.
Run the model with metrics collection bracketing the inference.
The signature keys of a model, in index order.
profile/1 folded into per-operator totals.
Types
-type accelerator() :: cpu | gpu | npu.
-type detail_level() :: 0..2147483647.
How many recent events profile/2 answers with, zero meaning all of them.
-type event() :: #{tag := binary(), us := non_neg_integer(), type := event_type(), source := event_source()}.
-type event_limit() :: 0..2147483647.
-type event_source() :: litert | tflite_interpreter | tflite_delegate | integer().
-type event_type() :: default | operator_kind() | runtime_instrumentation | telemetry | telemetry_report_settings | telemetry_delegate | telemetry_delegate_report_settings | integer().
-type operator_kind() :: operator | delegate_operator | delegate_profiled.
What a profiling event is. An integer means this build met a type LiteRT has since added, which is reported rather than dropped.
-type opts() :: #{accelerators => [accelerator()], precision => precision(), profile => boolean(), signature => signature_index() | binary() | string(), max_model_bytes => non_neg_integer()}.
-type precision() :: default | fp16 | fp32 | fp16_with_fp32_accum.
The signature index crosses into the NIF through enif_get_int, so the range is the C int range and not every non_neg_integer().
-type signature_index() :: 0..2147483647.
Same range and for the same reason: it crosses into the NIF as a C int.
-type summary_entry() :: #{tag := binary(), kind := operator_kind(), count := pos_integer(), us := non_neg_integer()}.
Whatever the accelerator chose to report. unsupported is a value of a kind LiteRtAny can hold but this library has no term for.
Functions
-spec available() -> boolean().
Whether this build has the LiteRT API at all.
It is a build option and it is off by default, so every other function in this module answers {error, <<"the LiteRT API was not compiled...">>} on an ordinary build. This is the cheap way to ask first rather than to find out from a call that was supposed to do something.
Which process this model belongs to, or undefined if nobody has claimed it. Answers {error, _} while another caller is inside the model, because the answer is read under the same lock that call holds.
Give the model to Pid, after which every other process is refused.
Unclaimed is the default and stays the default: a model nobody has claimed is usable from wherever its reference reaches, which is what this module is for. Claiming is how tflite_beam_litert_compiled_model_server makes its promise enforced rather than a convention, and it is available here for anyone building their own owner. A claim whose process has died is released, so a model is never stranded.
An environment with no directory to look for accelerator plugins in.
Not a guarantee that none is found: LiteRT still passes a bare filename to the platform loader, which has its own search paths. It is the way to say "I am not pointing you at one", not the way to say "there is none".
An environment that looks for accelerator plugins in Dir.
A GPU accelerator is a shared library loaded at run time, and LiteRT looks for it by filename relative to this directory. Leave it empty and the search happens relative to nothing, so a GPU compile falls back to the CPU with only a line in the log to say why.
Whether anything is left for the ordinary interpreter to run.
True means no undelegated operations remain, which several accelerators between them can satisfy just as well as one. False does not have to mean a split between accelerators either: it means at least one operation is running the ordinary way. profile/1 is what says which.
-spec io_sizes(reference()) -> {ok, {[non_neg_integer()], [non_neg_integer()]}} | {error, binary()}.
The byte size of each input and output buffer.
A compiled model on the CPU, with no profiling.
Compile Path in Env.
Keyword Parameters
-spec pending_events(reference()) -> {ok, non_neg_integer()} | {error, binary()}.
How many profiling events are waiting, without reading them.
Zero for a model built without profile => true. This is what sizes the copy profile/2 has to make, so it is the number to look at before calling it on a memory-constrained target.
What this build of the library can reach.
Compile-time answers, decided by LiteRT from the macros it was built with and by what this build turns off. opencl reads false everywhere here, because this library defines LITERT_DISABLE_OPENCL_SUPPORT: LiteRT's own OpenCL layer exists so it can pass CL buffers around itself and nothing here asks it to, while a GPU accelerator plugin brings its own OpenCL and is unaffected. This says "was it compiled in", not "is a device present"; the second question is answered by asking for the accelerator and being refused.
Every profiling event recorded so far, oldest first.
Empty unless the model was built with profile => true. Each event is a map of tag, us, type and source.
type and source are named against LiteRT's enumeration constants rather than passed through as numbers, so an upstream renumbering cannot quietly change what one means; a type this build has no name for arrives as its number. Telemetry events are included and carry a sentinel in place of a duration, which is why summarise_profile/1 drops them.
The buffer behind this is fixed and large: a compiled model asks for 512 * 1024 entries in LiteRT 2.2.0, not the 10240 a bare profiler defaults to. So a long-lived model does not grow without bound, but neither will it overflow soon. Reset it when you want to measure a change rather than to keep it in check.
-spec profile(reference(), event_limit()) -> {ok, [event()]} | {error, binary()}.
The most recent Limit profiling events, oldest first, or all of them when Limit is zero.
Worth using on a long-lived model. Events accumulate until the buffer's 512 * 1024 entries are full, and asking for all of them then builds half a million maps in one call.
Limit bounds the maps, not the reading. LiteRT will not hand over part of a backlog, so every call copies whatever pending_events/1 reports, at 104 bytes an event and twice over while LiteRT builds its own copy: a full buffer is about 109 MiB no matter how few events were asked for. That is nothing on a workstation and fatal on a board with 256 MB, so on a small target read often or call reset_profile/1, and check pending_events/1 rather than discovering it.
Forget the events recorded so far and keep recording.
Run the model over Inputs and return its outputs.
One binary per input, each exactly the size io_sizes/1 reports; anything else is refused rather than truncated or padded.
-spec run_with_metrics(reference(), [binary()]) -> {ok, {[binary()], [{binary(), metric_value()}]}} | {error, binary()}.
Run the model and collect whatever counters the accelerator reports.
-spec run_with_metrics(reference(), [binary()], detail_level()) -> {ok, {[binary()], [{binary(), metric_value()}]}} | {error, binary()}.
Run the model with metrics collection bracketing the inference.
Usually the counters come back empty. Filling them in is the accelerator's job, through two entries of its definition that are allowed to be null, so an empty list means nobody offered anything rather than that something went wrong. Every accelerator this library has been run against returned nothing; what a vendor backend does is its own business and is not established here.
Collection brackets the run rather than being started and stopped on its own, because a backend asked to report on an interval containing no inference has nothing to report on. Use profile/1 for timings; this is for counters a backend chooses to expose.
The signature keys of a model, in index order.
A model with no named signatures still has one, and it comes back as the empty key. Reading these needs the model but not a compile, so it is the cheap way to find out what new/3 can be asked for.
-spec summarise_profile(reference()) -> {ok, [summary_entry()]} | {error, binary()}.
profile/1 folded into per-operator totals.
Returns maps of tag, kind, count and us, sorted slowest first, over operator events only. A map rather than a tuple so that a field can be added later without breaking every caller that matched on position.
The events LiteRT records are nested. An Invoke encloses the operators it ran, and AllocateTensors and LiteRT's own buffer handling sit beside them, so folding everything together would count operators twice and put Invoke at the top of a list that claims to name the slowest operator. Only operator events are folded here; everything else is still in profile/1.
Nesting does not stop there, which is why Kind is in the tuple rather than summed away. A delegate_operator is an operator *inside* a delegate, and its time can already be counted in the enclosing delegate_profiled entry for the fused operation. LiteRT's own summariser keeps the two apart for the same reason. Totals within one Kind are additive; totals across kinds are not.