erltoken

Erlang token budgeting backed by the Rust tiktoken-rs crate.

Version: 0.3.0

All string inputs and outputs are binaries.

Versioning

bump2version patch
bump2version minor
bump2version major

Version bumps update the README, Erlang .app.src, and Rust package metadata together.

Build

rebar3 compile

Examples

1> erltoken:encoding_for_model(<<"gpt-4o">>).
{ok,<<"o200k_base">>}

2> erltoken:count(<<"gpt-4o">>, <<"hello world">>).
{ok,2}

3> erltoken:fits(<<"gpt-4o">>, <<"hello world">>, 10).
{ok,true}

4> erltoken:trim_to_token_limit(<<"cl100k_base">>, <<"hello world">>, 1).
{ok,<<"hello">>}

API

• list_encodings/0
• encoding_for_model/1
• count/2
• encode/2
• encode_with_special_tokens/2
• decode/2
• fits/3
• remaining/3
• trim_to_token_limit/3

For token APIs, the first argument may be an encoding name such as <<"cl100k_base">> or a model name such as <<"gpt-4o">>.

License

Apache-2.0