LemonAi.CallDispatcher (lemon_ai v0.1.0)

View Source

Central dispatcher for routing AI provider requests through rate limiting and circuit breaking infrastructure.

Overview

This module acts as a coordination point that:

  1. Checks the circuit breaker state
  2. Acquires a rate limit permit
  3. Enforces per-provider concurrency limits
  4. Returns appropriate errors when conditions aren't met

Usage

# Dispatch a request (typically wrapping an actual provider call)
case LemonAi.CallDispatcher.dispatch(:anthropic, fn -> make_api_call() end) do
  {:ok, result} -> handle_result(result)
  {:error, :rate_limited} -> retry_later()
  {:error, :circuit_open} -> use_fallback()
  {:error, :max_concurrency} -> queue_request()
  {:error, reason} -> handle_error(reason)
end

Configuration

Concurrency caps are configured per-provider via set_concurrency_cap/2. Default cap is 10 concurrent requests per provider.

Summary

Types

provider()
state()

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

dispatch(provider, callback)

Dispatch a request through rate limiting and circuit breaking.

get_active_requests(provider)

Get the number of active requests for a provider.

get_concurrency_cap(provider)

Get the current concurrency cap for a provider.

get_state()

Get dispatcher state for debugging/monitoring.

set_concurrency_cap(provider, cap)

Set the concurrency cap for a provider.

start_link(opts \\ [])

Start the call dispatcher.

Types

provider()

@type provider() :: atom()

state()

@type state() :: %{
  concurrency_caps: %{required(provider()) => pos_integer()},
  active_requests: %{required(provider()) => non_neg_integer()},
  monitors: %{required(reference()) => provider()}
}

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

dispatch(provider, callback)

@spec dispatch(provider(), (-> result)) :: result | {:error, atom()}
when result: any()

Dispatch a request through rate limiting and circuit breaking.

Returns the callback result, or an error if the request was blocked by rate limiting, circuit breaker, or concurrency limits.

Examples

LemonAi.CallDispatcher.dispatch(:anthropic, fn ->
  LemonAi.Providers.Anthropic.call(params)
end)

get_active_requests(provider)

@spec get_active_requests(provider()) :: non_neg_integer()

Get the number of active requests for a provider.

get_concurrency_cap(provider)

@spec get_concurrency_cap(provider()) :: pos_integer()

Get the current concurrency cap for a provider.

get_state()

@spec get_state() :: map()

Get dispatcher state for debugging/monitoring.

set_concurrency_cap(provider, cap)

@spec set_concurrency_cap(provider(), pos_integer()) :: :ok

Set the concurrency cap for a provider.

start_link(opts \\ [])

@spec start_link(keyword()) :: GenServer.on_start()

Start the call dispatcher.