budoux.gleam

A Gleam port of BudouX, a standalone, small, and language-neutral phrase segmenter that provides beautiful and legible line breaks.

Line breaks of a Japanese sentence without and with budoux

Install

gleam add budoux

Works on both the Erlang and JavaScript targets.

Usage

import budoux

pub fn main() {
  let parser = budoux.ja()

  budoux.parse(parser, "今日は天気です。")
  // -> ["今日は", "天気です。"]
}

API

budoux

Function Description
parse(parser: Parser, sentence: String) -> List(String) Splits a sentence into phrases.
ja() -> Parser Creates a parser with the Japanese model.
ja_word() -> Parser Creates a parser with the Japanese word model, which splits sentences into words.
zh_hans() -> Parser Creates a parser with the Simplified Chinese model.
zh_hant() -> Parser Creates a parser with the Traditional Chinese model.
th() -> Parser Creates a parser with the Thai model.
new(model: Model) -> Parser Creates a parser from a model.

budoux/model

Item Description
Model The scores of a BudouX model, grouped in the same way as its JSON file.
empty: Model A model with no scores.
decoder() -> decode.Decoder(Model) Decodes a model from the JSON format used by BudouX.

Development

The tools are managed in flake.nix, and the versions used in CI in .tool-versions.

Generated files

These files are generated from the BudouX release pinned in flake.nix, and the update workflow regenerates them.

Generated file Source in BudouX Script
src/budoux/model/*.gleam budoux/models/*.json codegen/models.nu
test/budoux/quality_cases.gleam tests/quality/ja.tsv codegen/quality_cases.nu

Acknowledgements

The algorithm, the pretrained models, and the test cases come from google/budoux.

License

Apache License 2.0