erltoken

View Source

CI

Erlang token budgeting and cost estimation backed by the Rust tiktoken crate.

Version: 0.1.7

All string inputs and outputs are binaries.

Versioning

bump2version patch
bump2version minor
bump2version major

Version bumps update the README, Erlang .app.src, and Rust package metadata together.

Build

rebar3 compile

Examples

1> erltoken:encoding_for_model(<<"gpt-4o">>).
{ok,<<"o200k_base">>}

2> erltoken:count(<<"gpt-4o">>, <<"hello world">>).
{ok,2}

3> erltoken:fits(<<"gpt-4o">>, <<"hello world">>, 10).
{ok,true}

4> erltoken:trim_to_token_limit(<<"cl100k_base">>, <<"hello world">>, 1).
{ok,<<"hello">>}

5> erltoken:estimate_cost_usd_micro(<<"gpt-4o">>, 1000, 1000).
{ok,12500}

estimate_cost_usd_micro/3 returns micro-USD. 12500 means $0.012500.

API

  • list_encodings/0
  • encoding_for_model/1
  • count/2
  • encode/2
  • encode_with_special_tokens/2
  • decode/2
  • fits/3
  • remaining/3
  • trim_to_token_limit/3
  • estimate_cost_usd_micro/3

For token APIs, the first argument may be an encoding name such as <<"cl100k_base">> or a model name such as <<"gpt-4o">>.

License

Apache-2.0