budoux.gleam
A Gleam port of BudouX, a standalone, small, and language-neutral phrase segmenter that provides beautiful and legible line breaks.
Install
gleam add budoux
Works on both the Erlang and JavaScript targets.
Usage
import budoux
pub fn main() {
let parser = budoux.ja()
budoux.parse(parser, "今日は天気です。")
// -> ["今日は", "天気です。"]
}
API
budoux
| Function | Description |
|---|---|
parse(parser: Parser, sentence: String) -> List(String) |
Splits a sentence into phrases. |
ja() -> Parser |
Creates a parser with the Japanese model. |
ja_word() -> Parser |
Creates a parser with the Japanese word model, which splits sentences into words. |
zh_hans() -> Parser |
Creates a parser with the Simplified Chinese model. |
zh_hant() -> Parser |
Creates a parser with the Traditional Chinese model. |
th() -> Parser |
Creates a parser with the Thai model. |
new(model: Model) -> Parser |
Creates a parser from a model. |
budoux/model
| Item | Description |
|---|---|
Model |
The scores of a BudouX model, grouped in the same way as its JSON file. |
empty: Model |
A model with no scores. |
decoder() -> decode.Decoder(Model) |
Decodes a model from the JSON format used by BudouX. |
Development
The tools are managed in flake.nix, and the versions used in CI in .tool-versions.
Generated files
These files are generated from the BudouX release pinned in flake.nix, and the update workflow regenerates them.
| Generated file | Source in BudouX | Script |
|---|---|---|
src/budoux/model/*.gleam |
budoux/models/*.json |
codegen/models.nu |
test/budoux/quality_cases.gleam |
tests/quality/ja.tsv |
codegen/quality_cases.nu |
Acknowledgements
The algorithm, the pretrained models, and the test cases come from google/budoux.