LLMProxy.Providers.Routing.Performance (llm_proxy v0.2.0)

Keeps bounded, per-deployment performance observations for latency-aware routing.

State is intentionally node-local and ephemeral. Durable request accounting remains in storage, while this process only serves the routing hot path.

Summary

Types

stats()

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

observe(attempt, sample, event \\ nil)

order(model, deployments, request)

reset()

start_link(opts)

stats(attempt, request)

Types

stats()

@type stats() :: %{
  success_count: non_neg_integer(),
  ttft_count: non_neg_integer(),
  error_count: non_neg_integer(),
  median_duration_ms: number() | nil,
  median_ttft_ms: number() | nil,
  updated_at: integer() | nil
}

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

observe(attempt, sample, event \\ nil)

@spec observe(
  LLMProxy.Providers.Attempt.t(),
  LLMProxy.Providers.Routing.Sample.t(),
  atom() | nil
) ::
  :ok

order(model, deployments, request)

@spec order(
  String.t(),
  [LLMProxy.Catalog.Deployment.t()],
  LLMProxy.Protocol.Request.t()
) :: [
  LLMProxy.Catalog.Deployment.t()
]

reset()

@spec reset() :: :ok

start_link(opts)

@spec start_link(keyword()) :: GenServer.on_start()

stats(attempt, request)

@spec stats(LLMProxy.Providers.Attempt.t(), LLMProxy.Protocol.Request.t()) :: stats()