Nous.Eval.YamlLoader (nous v0.15.8)

View Source

Loads evaluation suites and test cases from YAML files.

Example YAML Format

name: my_suite
description: Test suite for weather agent
default_model: lmstudio:ministral-3-14b-reasoning
default_timeout: 30000
parallelism: 2

test_cases:
  - id: basic_query
    name: Basic Weather Query
    input: "What's the weather in Tokyo?"
    expected:
      contains: ["weather", "Tokyo"]
    eval_type: contains
    tags: [basic, weather]
    timeout: 20000

  - id: exact_response
    name: Math Test
    input: "What is 2+2?"
    expected: "4"
    eval_type: exact_match

  - id: quality_check
    name: Explanation Quality
    input: "Explain recursion"
    expected:
      criteria: "Is the explanation clear?"
    eval_type: llm_judge
    eval_config:
      judge_model: lmstudio:ministral-3-14b-reasoning

Summary

Functions

load_directory(dir)

Load all suites from a directory.

load_suite(path)

Load a suite from a YAML file.

Functions

load_directory(dir)

@spec load_directory(String.t()) :: {:ok, [Nous.Eval.Suite.t()]} | {:error, term()}

Load all suites from a directory.

load_suite(path)

@spec load_suite(String.t()) :: {:ok, Nous.Eval.Suite.t()} | {:error, term()}

Load a suite from a YAML file.