ExLLM.Adapters.Shared.VisionFormatter behaviour (ex_llm v0.5.0)

View Source

Unified vision/multimodal content formatting for LLM providers.

This module standardizes the handling of images and other multimodal content across different providers, reducing code duplication.

Features:

  • Image format validation and detection
  • Base64 encoding/decoding
  • Provider-specific formatting
  • URL and file handling
  • Content type detection

Summary

Callbacks

Callback for provider-specific vision content formatting.

Callback to check if a model supports vision.

Functions

Create a base64 image content part.

Detect media type from file extension or magic bytes.

Format messages containing vision content for a specific provider.

Check if messages contain vision content.

Create an image URL content part.

Load an image from a file path and encode it.

Check if a single message has vision content.

Validate that an image format is supported.

Callbacks

format_vision_content(content)

(optional)
@callback format_vision_content(content :: map()) :: map()

Callback for provider-specific vision content formatting.

model_supports_vision?(model)

(optional)
@callback model_supports_vision?(model :: String.t()) :: boolean()

Callback to check if a model supports vision.

Functions

base64_image(data, media_type, options \\ [])

@spec base64_image(binary(), String.t(), keyword()) :: map()

Create a base64 image content part.

detect_media_type(path, data)

@spec detect_media_type(String.t(), binary()) ::
  {:ok, String.t()} | {:error, :unknown_format}

Detect media type from file extension or magic bytes.

format_messages(messages, provider)

@spec format_messages([map()], atom()) :: [map()]

Format messages containing vision content for a specific provider.

Examples

messages = [
  %{
    role: "user",
    content: [
      %{type: "text", text: "What's in this image?"},
      %{type: "image_url", image_url: %{url: "https://example.com/cat.jpg"}}
    ]
  }
]

formatted = VisionFormatter.format_messages(messages, :anthropic)

has_vision_content?(messages)

@spec has_vision_content?([map()]) :: boolean()

Check if messages contain vision content.

image_url(url, options \\ [])

@spec image_url(
  String.t(),
  keyword()
) :: map()

Create an image URL content part.

load_image_file(path, options \\ [])

@spec load_image_file(
  String.t(),
  keyword()
) :: {:ok, map()} | {:error, term()}

Load an image from a file path and encode it.

Returns a content part ready for inclusion in a message.

Options

  • :format - Force output format (:base64 or :url)
  • :max_size - Maximum file size in bytes
  • :detail - Image detail level ("low", "high", "auto")

message_has_vision_content?(arg1)

@spec message_has_vision_content?(map()) :: boolean()

Check if a single message has vision content.

validate_media_type(media_type)

@spec validate_media_type(String.t()) :: :ok | {:error, :unsupported_format}

Validate that an image format is supported.