This module provides an implementation of the OpenAI evals API. The API reference can be found at https://platform.openai.com/docs/api-reference/evals.
Summary
Functions
Cancels a specific run for an evaluation.
Calls the eval 'create' endpoint.
Creates a new run for an evaluation.
Calls the eval delete endpoint.
Deletes a specific run for an evaluation.
Retrieves a specific run for an evaluation.
Retrieves a specific output item for a run.
Lists all runs for a specific evaluation.
Lists all evals that belong to the user's organization.
Lists all output items for a specific run.
Creates a new eval request
Creates a new eval run request
Calls the eval update endpoint.
Functions
Cancels a specific run for an evaluation.
See https://platform.openai.com/docs/api-reference/evals/cancelRun for more information.
Calls the eval 'create' endpoint.
See https://platform.openai.com/docs/api-reference/evals/create for more information.
Creates a new run for an evaluation.
See https://platform.openai.com/docs/api-reference/evals/createRun for more information.
Calls the eval delete endpoint.
See https://platform.openai.com/docs/api-reference/evals/delete for more information.
Deletes a specific run for an evaluation.
See https://platform.openai.com/docs/api-reference/evals/deleteRun for more information.
Retrieves a specific run for an evaluation.
See https://platform.openai.com/docs/api-reference/evals/getRun for more information.
Retrieves a specific output item for a run.
See https://platform.openai.com/docs/api-reference/evals/getRunOutputItem for more information.
Lists all runs for a specific evaluation.
See https://platform.openai.com/docs/api-reference/evals/getRuns for more information.
Lists all evals that belong to the user's organization.
See https://platform.openai.com/docs/api-reference/evals/list for more information.
Lists all output items for a specific run.
See https://platform.openai.com/docs/api-reference/evals/listRunOutputItems for more information.
Creates a new eval request
See https://platform.openai.com/docs/api-reference/evals/create for full list of request parameters.
Example usage:
iex> _request = OpenaiEx.Evals.new(name: "Sentiment")
%{name: "Sentiment"}
iex> _request = OpenaiEx.Evals.new(%{name: "Sentiment", testing_criteria: [%{type: "label_model", ...}]})
%{name: "Sentiment", testing_criteria: [%{type: "label_model", ...}]}Creates a new eval run request
See https://platform.openai.com/docs/api-reference/evals/createRun for full list of request parameters.
Example usage:
iex> _request = OpenaiEx.Evals.new_run(name: "gpt-4o-mini")
%{name: "gpt-4o-mini"}
iex> _request = OpenaiEx.Evals.new_run(%{name: "gpt-4o-mini", data_source: %{type: "completions", ...}})
%{name: "gpt-4o-mini", data_source: %{type: "completions", ...}}Calls the eval update endpoint.
See https://platform.openai.com/docs/api-reference/evals/update for more information.