Gpt3Tokenizer

BPE Encoder Decoder for GPT-3 implemented in native Elixir.

Installation

The package can be installed by adding gpt3_tokenizer to your list of dependencies in mix.exs:

def deps do
  [
    {:gpt3_tokenizer, "~> 0.1.0"}
  ]
end

Usage

Gpt3Tokenizer.encode("hello 👋 world 🌍")
Gpt3Tokenizer.decode([31373, 50169, 233, 995, 12520, 234, 235])
Gpt3Tokenizer.token_count("This sentence is 6 tokens long")

Reference

• OpenAI official online tokenizer
• gpt-3-encoder

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/gpt3_tokenizer.