LLMDB.Normalize (LLM DB v2026.8.3)

Copy Markdown View Source

Complete normalization utilities for raw data into consistent formats.

This module handles ALL normalization in one place:

  • Provider IDs: string → atom (with separator → underscore conversion)
  • Model providers: string → atom
  • Modalities: string → atom (from valid set)
  • Tags: map → list, nil → []
  • Dates: DateTime/Date → ISO8601 string
  • Removing nil values from maps

Resolves packaged providers through a generated registry and uses String.to_existing_atom/1 for trusted custom providers. Uses String.to_atom/1 only in unsafe mode during build-time Mix tasks.

Summary

Functions

normalize_date(date_string)

Normalizes a date string to "YYYY-MM-DD" format.

normalize_model_identity(model, opts \\ [])

Normalizes a model's identity to a {provider_atom, model_id} tuple.

normalize_models(models)

Normalizes a list of model maps.

normalize_provider_id(provider_id, opts \\ [])

Normalizes a provider ID to an atom.

normalize_providers(providers)

Normalizes a list of provider maps.

Functions

normalize_date(date_string)

@spec normalize_date(String.t() | nil) :: String.t() | nil

Normalizes a date string to "YYYY-MM-DD" format.

Attempts to parse and normalize various date formats. If the date cannot be normalized, it is returned as-is.

Examples

iex> LLMDB.Normalize.normalize_date("2024-01-15")
"2024-01-15"

iex> LLMDB.Normalize.normalize_date("2024/01/15")
"2024-01-15"

iex> LLMDB.Normalize.normalize_date("invalid-date")
"invalid-date"

iex> LLMDB.Normalize.normalize_date(nil)
nil

normalize_model_identity(model, opts \\ [])

@spec normalize_model_identity(
  map(),
  keyword()
) :: {:ok, {atom(), String.t()}} | {:error, term()}

Normalizes a model's identity to a {provider_atom, model_id} tuple.

Extracts the provider (as an atom) and id from a model map.

Examples

iex> LLMDB.Normalize.normalize_model_identity(%{provider: "google-vertex", id: "gemini-pro"})
{:ok, {:google_vertex, "gemini-pro"}}

iex> LLMDB.Normalize.normalize_model_identity(%{provider: :openai, id: "gpt-4"})
{:ok, {:openai, "gpt-4"}}

iex> LLMDB.Normalize.normalize_model_identity(%{provider: "openai"})
{:error, :missing_id}

normalize_models(models)

@spec normalize_models([map()]) :: [map()]

Normalizes a list of model maps.

Applies normalize_provider_id to the :provider field and ensures :id is present.

Examples

iex> LLMDB.Normalize.normalize_models([%{provider: "google-vertex", id: "gemini-pro"}])
[%{provider: :google_vertex, id: "gemini-pro"}]

normalize_provider_id(provider_id, opts \\ [])

@spec normalize_provider_id(
  binary() | atom(),
  keyword()
) :: {:ok, atom()} | {:error, :bad_provider | :unknown_provider}

Normalizes a provider ID to an atom.

Converts binary provider IDs to atoms, handling hyphens and dots by converting them to underscores. At runtime, it accepts providers from the generated package registry or atoms that trusted custom configuration already created. During build-time tasks, unsafe conversion is allowed.

Examples

iex> LLMDB.Normalize.normalize_provider_id("google-vertex")
{:ok, :google_vertex}

iex> LLMDB.Normalize.normalize_provider_id(:openai)
{:ok, :openai}

iex> LLMDB.Normalize.normalize_provider_id("maliciousaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")
{:error, :bad_provider}

normalize_providers(providers)

@spec normalize_providers([map()]) :: [map()]

Normalizes a list of provider maps.

Applies normalize_provider_id to the :id field of each provider map.

Examples

iex> LLMDB.Normalize.normalize_providers([%{id: "google-vertex"}, %{id: :openai}])
[%{id: :google_vertex}, %{id: :openai}]