Nous.Eval.Optimizer.Strategies.Bayesian (nous v0.16.0)

View Source

Bayesian optimization strategy inspired by TPE (Tree-structured Parzen Estimator).

This strategy learns from previous trials to focus exploration on promising regions of the search space. It balances exploration (trying new areas) with exploitation (refining good areas).

How It Works

  1. Start with random samples to build initial knowledge
  2. Split trials into "good" (above threshold) and "bad" (below threshold)
  3. Model the distribution of good vs bad configurations
  4. Sample new configurations more likely to be in "good" regions
  5. Repeat until budget exhausted

Options

  • :n_trials - Total number of trials (default: 100)
  • :n_initial - Initial random trials before optimization (default: 10)
  • :gamma - Quantile for splitting good/bad (default: 0.25 = top 25%)
  • :timeout - Total timeout in ms (default: 3600000)
  • :early_stop - Stop if score reaches threshold
  • :verbose - Print progress (default: true)

Example

Optimizer.optimize(suite, params,
  strategy: :bayesian,
  n_trials: 50,
  n_initial: 10,
  metric: :score
)

When to Use

Bayesian optimization is recommended when:

  • Evaluations are expensive (LLM API calls cost time/money)
  • You want to find good configurations with fewer trials
  • The search space is continuous or mixed
  • Some parameter combinations are likely better than others

Limitations

  • More complex than random search
  • May get stuck in local optima
  • Requires enough initial samples to build good model