ToonEx

TOON (Token-Oriented Object Notation) encoder and decoder for Elixir and Phoenix.

TOON is a compact data format optimized for LLM token efficiency, achieving 30-60% token reduction compared to JSON while maintaining readability.

TOON is suit for LLM, I try to apply to replace JSON in Phoenix.

Features

• 🎯 Token Efficient: 30-60% fewer tokens than JSON
• 📖 Human Readable: Indentation-based structure like YAML
• 🔧 Three Array Formats: Inline, tabular, and list formats
• ✅ Spec Compliant: Tested against official TOON v1.3 specification
• 🛡️ Type Safe: Full Dialyzer support with comprehensive typespecs
• 🔌 Protocol Support: Custom encoding via ToonEx.Encoder protocol
• 📊 Telemetry: Built-in instrumentation for monitoring
• 🛠️ Convertor: Support convert between JSON & TOON

Installation

Add toon_ex to your list of dependencies in mix.exs:

def deps do
  [
    {:toon_ex, "~> 0.1.0"}
  ]
end

Quick Start

Encoding

# Simple object
ToonEx.encode!(%{"name" => "Alice", "age" => 30})
# => "age: 30\\nname: Alice"

# Nested object
ToonEx.encode!(%{"user" => %{"name" => "Bob"}})
# => "user:\\n  name: Bob"

# Arrays
ToonEx.encode!(%{"tags" => ["elixir", "toon"]})
# => "tags[2]: elixir,toon"

Decoding

ToonEx.decode!("name: Alice\\nage: 30")
# => %{"name" => "Alice", "age" => 30}

ToonEx.decode!("tags[2]: a,b")
# => %{"tags" => ["a", "b"]}

# With options
ToonEx.decode!("user:\\n    name: Alice", indent_size: 4)
# => %{"user" => %{"name" => "Alice"}}

Comprehensive Examples

Primitives

ToonEx.encode!(nil)            # => "null"
ToonEx.encode!(true)           # => "true"
ToonEx.encode!(42)             # => "42"
ToonEx.encode!(3.14)           # => "3.14"
ToonEx.encode!("hello")        # => "hello"
ToonEx.encode!("hello world")  # => "\\"hello world\\"" (auto-quoted)

Objects

# Simple objects
ToonEx.encode!(%{"name" => "Alice", "age" => 30})
# =>
# age: 30
# name: Alice

# Nested objects
ToonEx.encode!(%{
  "user" => %{
    "name" => "Bob",
    "email" => "bob@example.com"
  }
})
# =>
# user:
#   email: bob@example.com
#   name: Bob

Arrays

# Inline arrays (primitives)
ToonEx.encode!(%{"tags" => ["elixir", "toon", "llm"]})
# => "tags[3]: elixir,toon,llm"

# Tabular arrays (uniform objects)
ToonEx.encode!(%{
  "users" => [
    %{"name" => "Alice", "age" => 30},
    %{"name" => "Bob", "age" => 25}
  ]
})
# => "users[2]{age,name}:\\n  30,Alice\\n  25,Bob"

# List-style arrays (mixed or nested)
ToonEx.encode!(%{
  "items" => [
    %{"type" => "book", "title" => "Elixir Guide"},
    %{"type" => "video", "duration" => 120}
  ]
})
# => "items[2]:\\n  - duration: 120\\n    type: video\\n  - title: \\"Elixir Guide\\"\\n    type: book"

Encoding Options

# Custom delimiters
ToonEx.encode!(%{"tags" => ["a", "b", "c"]}, delimiter: "\\t")
# => "tags[3\\t]: a\\tb\\tc"

ToonEx.encode!(%{"values" => [1, 2, 3]}, delimiter: "|")
# => "values[3|]: 1|2|3"

# Length markers
ToonEx.encode!(%{"tags" => ["a", "b", "c"]}, length_marker: "#")
# => "tags[#3]: a,b,c"

# Custom indentation
ToonEx.encode!(%{"user" => %{"name" => "Alice"}}, indent: 4)
# => "user:\\n    name: Alice"

Decoding Options

# Atom keys
ToonEx.decode!("name: Alice", keys: :atoms)
# => %{name: "Alice"}

# Custom indent size
ToonEx.decode!("user:\\n    name: Alice", indent_size: 4)
# => %{"user" => %{"name" => "Alice"}}

# Strict mode (default: true)
ToonEx.decode!("  name: Alice", strict: false)  # Accepts non-standard indentation
# => %{"name" => "Alice"}

Specification Compliance

This implementation is tested against the official TOON specification v1.3.

Testing Approach

Tests use semantic equivalence checking: both encoder output and expected output are decoded and compared. This ensures correctness while accommodating Elixir 1.19's automatic map key sorting (outputs may differ in key order but decode to identical data structures).

Limitation

Not support for validated TOON in current version.

Not optimized for high traffic application yet.

Testing

The test suite uses official TOON specification fixtures:

# Run all tests against official spec fixtures
mix test

# Run only fixture-based tests
mix test test/toon_ex/fixtures_test.exs

Test fixtures are loaded from the toon-format/spec repository via git submodule.

TOON Specification

This implementation follows TOON Specification v1.3.

Contributing

Contributions are welcome!

Author

Created by Kentaro Kuribayashi

Forked and updated by Manh Vu

License

MIT License - see LICENSE.