A GenServer that wraps a loaded TorchScript model for concurrent serving.

Provides process isolation, fault tolerance, and serialized access to model inference. Forward calls are serialized through the GenServer to ensure thread safety for models with mutable state (e.g., BatchNorm, Dropout).

Telemetry Events

The server emits the following :telemetry events:

• [:extorch, :jit, :load, :start] - When model loading begins.

  • Measurements: %{system_time: integer}
  • Metadata: %{path: String.t(), device: atom()}

• [:extorch, :jit, :load, :stop] - When model loading completes.

  • Measurements: %{duration: native_time}
  • Metadata: %{path: String.t(), device: atom()}

• [:extorch, :jit, :load, :exception] - When model loading fails.

  • Measurements: %{duration: native_time}
  • Metadata: %{path: String.t(), device: atom(), kind: atom(), reason: term()}

• [:extorch, :jit, :forward, :start] - When inference begins.

  • Measurements: %{system_time: integer}
  • Metadata: %{path: String.t(), device: atom(), input_count: integer()}

• [:extorch, :jit, :forward, :stop] - When inference completes.

  • Measurements: %{duration: native_time}
  • Metadata: %{path: String.t(), device: atom(), input_count: integer()}

• [:extorch, :jit, :forward, :exception] - When inference fails.

  • Measurements: %{duration: native_time}
  • Metadata: %{path: String.t(), device: atom(), input_count: integer(), kind: atom(), reason: term()}

Example

{:ok, pid} = ExTorch.JIT.Server.start_link(path: "model.pt", device: :cpu)
result = ExTorch.JIT.Server.predict(pid, [input_tensor])