OpenAI Provider

Embedding generation using OpenAI's Embeddings API.

Requirements

• OpenAI API key

Configuration

{ok, State} = barrel_embed:init(#{
    embedder => {openai, #{
        api_key => <<"sk-...">>,                    % or use env var
        model => <<"text-embedding-3-small">>       % default
    }}
}).

Options

Using Environment Variable

Set OPENAI_API_KEY instead of passing in config:

export OPENAI_API_KEY=sk-...

{ok, State} = barrel_embed:init(#{
    embedder => {openai, #{}}  % uses env var
}).

Supported Models

Example

%% Initialize
{ok, State} = barrel_embed:init(#{
    embedder => {openai, #{
        api_key => <<"sk-proj-...">>,
        model => <<"text-embedding-3-small">>
    }}
}).

%% Generate embeddings
{ok, Vec} = barrel_embed:embed(<<"Machine learning is fascinating">>, State).

%% Batch (more efficient for multiple texts)
{ok, Vecs} = barrel_embed:embed_batch([
    <<"Document about AI">>,
    <<"Document about databases">>,
    <<"Document about networking">>
], State).

Rate Limiting

OpenAI has rate limits. For high-volume usage, consider:

1. Using batch embedding instead of single calls
2. Implementing retry logic at the application level
3. Using the provider chain with fallback

#{embedder => [
    {openai, #{}},
    {ollama, #{url => <<"http://localhost:11434">>}}  % fallback
]}

Cost Considerations

OpenAI charges per token. To optimize costs:

• Use text-embedding-3-small for most use cases
• Batch multiple texts in single API calls
• Cache embeddings when possible