Nous.Eval.Config (nous v0.15.6)

View Source

Configuration for the evaluation framework.

Configuration can be set via:

  • Application environment
  • Environment variables
  • Explicit options passed to functions

Application Configuration

config :nous, Nous.Eval,
  default_model: "lmstudio:ministral-3-14b-reasoning",
  default_timeout: 30_000,
  parallelism: 4,
  results_path: "priv/eval_results"

Environment Variables

NOUS_EVAL_DEFAULT_MODEL=lmstudio:model
NOUS_EVAL_DEFAULT_TIMEOUT=60000

Cost Configuration

Provider pricing for cost estimation (per 1K tokens):

config :nous, Nous.Eval,
  cost_config: %{
    "lmstudio" => %{input: 0.0, output: 0.0},
    "openai" => %{input: 0.01, output: 0.03},
    "anthropic" => %{input: 0.015, output: 0.075}
  }

Summary

Types

t()

Functions

estimate_cost(provider, input_tokens, output_tokens)

Estimate cost for token usage.

get(opts \\ [])

Get configuration from application environment and env vars.

get_model(test_case, suite, opts \\ [])

Get model for a test, with fallback chain.

lmstudio_config()

Get LM Studio configuration from environment.

Types

t()

@type t() :: %Nous.Eval.Config{
  cost_config: map(),
  default_instructions: String.t() | nil,
  default_model: String.t() | nil,
  default_timeout: non_neg_integer(),
  parallelism: non_neg_integer(),
  results_path: String.t(),
  store_results: boolean()
}

Functions

estimate_cost(provider, input_tokens, output_tokens)

@spec estimate_cost(String.t(), non_neg_integer(), non_neg_integer()) :: float()

Estimate cost for token usage.

Returns cost in USD.

get(opts \\ [])

@spec get(keyword()) :: t()

Get configuration from application environment and env vars.

Merges configuration from:

  1. Defaults
  2. Application config
  3. Environment variables
  4. Explicit options

get_model(test_case, suite, opts \\ [])

@spec get_model(Nous.Eval.TestCase.t() | nil, Nous.Eval.Suite.t() | nil, keyword()) ::
  String.t() | nil

Get model for a test, with fallback chain.

lmstudio_config()

@spec lmstudio_config() :: keyword()

Get LM Studio configuration from environment.