toon_codec

Package VersionHex DocsTestsLicense

A Gleam implementation of TOON (Token-Oriented Object Notation) - a compact, human-readable format designed to reduce token usage in Large Language Model (LLM) input.

Origin

tooncodec is a Gleam port of the original TOON format created by _Johann Schopplich → https://github.com/johannschopplich/toon/tree/main

About TOON

TOON is a serialization format optimized for LLM consumption that achieves significant token reduction compared to JSON while maintaining human readability. It uses indentation-based syntax, compact array notation, and eliminates redundant syntax like braces and quotes where possible.

Key Benefits

Installation

Add toon_codec to your Gleam project:

gleam add toon_codec

Quick Start

import toon_codec

import toon_codec/types.{Array, Bool, Number, Object, String}



pub fn main() {

  // Create a data structure

  let user = Object([

    #("name", String("Alice")),

    #("age", Number(30.0)),

    #("active", Bool(True)),

    #("tags", Array([String("admin"), String("developer")])),

  ])



  // Encode to TOON format

  let toon_str = toon_codec.encode(user)

  // Result:

  // name: Alice

  // age: 30

  // active: true

  // tags[2]: admin,developer



  // Decode back to JsonValue

  case toon_codec.decode(toon_str) {

    Ok(decoded) -> io.println("Successfully decoded!")

    Error(err) -> io.println("Error: " <> string.inspect(err))

  }

}

For a comprehensive working example with multiple use cases, see the src/example/ folder and run:

gleam run -m example/demo

Features

Usage Examples

Primitives

import toon_codec

import toon_codec/types.{Bool, Null, Number, String}



// Strings

toon_codec.encode(String("hello"))

// -> "hello"



// Numbers

toon_codec.encode(Number(42.5))

// -> "42.5"



// Booleans

toon_codec.encode(Bool(True))

// -> "true"



// Null

toon_codec.encode(Null)

// -> "null"

Objects

import toon_codec/types.{Number, Object, String}



// Simple object

let user = Object([

  #("name", String("Alice")),

  #("age", Number(30.0)),

])



toon_codec.encode(user)

// -> "name: Alice\nage: 30"



// Nested object

let person = Object([

  #("name", String("Bob")),

  #("address", Object([

    #("city", String("NYC")),

    #("zip", String("10001")),

  ])),

])



toon_codec.encode(person)

// -> "name: Bob\naddress:\n  city: NYC\n  zip: \"10001\""

Arrays

Inline Arrays (Primitives)

import toon_codec/types.{Array, Number}



let numbers = Array([Number(1.0), Number(2.0), Number(3.0)])

toon_codec.encode(numbers)

// -> "[3]: 1,2,3"

Tabular Arrays (Objects with Same Keys)

import toon_codec/types.{Array, Number, Object, String}



let users = Array([

  Object([#("name", String("Alice")), #("age", Number(30.0))]),

  Object([#("name", String("Bob")), #("age", Number(25.0))]),

])



toon_codec.encode(users)

// -> "[2]{name,age}:\n  Alice,30\n  Bob,25"

Expanded Arrays (Mixed Types)

import toon_codec/types.{Array, Number, Object, String}



let mixed = Array([

  String("item1"),

  Number(42.0),

  Object([#("key", String("value"))]),

])



toon_codec.encode(mixed)

// -> "[3]:\n  - item1\n  - 42\n  - key: value"

Custom Delimiters

import toon_codec

import toon_codec/types.{Array, EncodeOptions, Number, Tab}



let numbers = Array([Number(1.0), Number(2.0), Number(3.0)])

let options = EncodeOptions(indent: 2, delimiter: Tab, length_marker: types.NoMarker)



toon_codec.encode_with_options(numbers, options)

// -> "[3\t]: 1\t2\t3"

Custom Options

import toon_codec/types.{Comma, DecodeOptions, EncodeOptions, HashMarker}



// Encoding options

let encode_opts = EncodeOptions(

  indent: 4,                    // 4 spaces per indentation level

  delimiter: Comma,             // Use comma separator (Tab, Pipe also available)

  length_marker: HashMarker,    // Prefix array lengths with #: [#3]:

)



// Decoding options

let decode_opts = DecodeOptions(

  indent: 4,      // Match encoding indent

  strict: True,   // Validate array counts and indentation

)



let encoded = toon_codec.encode_with_options(value, encode_opts)

let result = toon_codec.decode_with_options(encoded, decode_opts)

API Reference

Core Functions

Types

TOON Format Examples

JSON vs TOON Comparison

JSON:

{

  "users": [

    { "name": "Alice", "age": 30, "active": true },

    { "name": "Bob", "age": 25, "active": false }

  ],

  "total": 2

}

TOON:

users[2]{name,age,active}:

  Alice,30,true

  Bob,25,false

total: 2

Token Savings: ~40% reduction in this example.

Testing

toon_codec includes a comprehensive test suite with 113 tests covering:

Run tests with:

gleam test

Specification

toon_codec implements the TOON Specification v1.2. For complete format details, see:

Development

# Clone the repository

git clone https://github.com/axelbellec/toon.git

cd toon



# Install dependencies

gleam deps download



# Run tests

gleam test



# Build the project

gleam build



# Format code

gleam format

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments