GenAI.Provider.LiteLLM (Noizu Labs, GenAI Wrapper v0.3.2)

Copy Markdown

Module for interacting with a LiteLLM proxy.

LiteLLM is a self-hosted, OpenAI-compatible proxy/gateway in front of many upstream providers, so it reuses the standard behaviour/encoder machinery. Its base_url is deployment-specific — set config :genai, :litellm, base_url: "...", api_key: "..." (defaults to http://localhost:4000). Auth is a Bearer "virtual key".

Media generation (image / speech / transcription / music / sfx / video) is handled by GenAI.Provider.LiteLLM.Media.

Summary

Functions

base_url()

Runtime base_url from config :genai, :litellm, base_url: (defaults to localhost:4000).

chat(messages, tools, settings)
chat(model, messages, tools, hyper_parameters, provider_settings \\ [], context \\ nil, options \\ nil)

Callback implementation for GenAI.InferenceProviderBehaviour.chat/7.

config_key()

Return config_key inference provide application config stored under :genai entry

default_encoder()
effective_settings(model, session, context, options \\ nil)

Obtain map of effective settings: settings, model_settings, provider_settings, config_settings, etc.

endpoint(model, settings, session, context, options \\ nil)

Prepare endpoint and method to make inference call to

generate_media(request, options)

Callback implementation for GenAI.InferenceProviderBehaviour.generate_media/2.

headers(options)
headers(model, settings, session, context, options \\ nil)

Prepare request headers

models(settings \\ [])

Retrieves a list of models the LiteLLM proxy exposes.

request_body(model, messages, tools, settings, session, context, options \\ nil)

Prepare request body to be passed to inference call.

run(session, context, options \\ nil)

Build and run inference thread

standardize_model(model)

Callback implementation for GenAI.InferenceProviderBehaviour.standardize_model/1.

stream(session, context, options \\ nil)

Build and run inference thread in streaming mode

supported_modalities()

Callback implementation for GenAI.InferenceProviderBehaviour.supported_modalities/0.

Functions

base_url()

Runtime base_url from config :genai, :litellm, base_url: (defaults to localhost:4000).

chat(messages, tools, settings)

chat(model, messages, tools, hyper_parameters, provider_settings \\ [], context \\ nil, options \\ nil)

Callback implementation for GenAI.InferenceProviderBehaviour.chat/7.

config_key()

Return config_key inference provide application config stored under :genai entry

default_encoder()

effective_settings(model, session, context, options \\ nil)

Obtain map of effective settings: settings, model_settings, provider_settings, config_settings, etc.

endpoint(model, settings, session, context, options \\ nil)

Prepare endpoint and method to make inference call to

generate_media(request, options)

Callback implementation for GenAI.InferenceProviderBehaviour.generate_media/2.

headers(options)

headers(model, settings, session, context, options \\ nil)

Prepare request headers

models(settings \\ [])

Retrieves a list of models the LiteLLM proxy exposes.

request_body(model, messages, tools, settings, session, context, options \\ nil)

Prepare request body to be passed to inference call.

run(session, context, options \\ nil)

Build and run inference thread

standardize_model(model)

Callback implementation for GenAI.InferenceProviderBehaviour.standardize_model/1.

stream(session, context, options \\ nil)

Build and run inference thread in streaming mode

supported_modalities()

Callback implementation for GenAI.InferenceProviderBehaviour.supported_modalities/0.