Mojibake for Elixir
Unicode text processing for Elixir: normalization, locale-aware casing, caseless matching, grapheme, word, sentence, and line segmentation, terminal width, character data from the Unicode Character Database, collation, internationalized domain names, confusable detection, identifiers, encoding conversion, BCP 47 language tags, and bidirectional text. It passes the official Unicode conformance tests.
It wraps Mojibake, a self-contained Unicode library written in
C11 with no dependencies, as a NIF. The package compiles the C sources when you build it, so all
you need is a C compiler and make: no CMake, no system library, and no data files to ship.
(Despite the name, it is not a library that repairs garbled text. Mojibake is the Japanese word for the garbage you get when text is decoded with the wrong encoding, which is what correct Unicode handling avoids.)
def deps do
[{:mojibake, "~> 0.8"}]
end
On Linux (x86_64 and aarch64, glibc 2.35 or newer), macOS (Apple silicon and Intel), and
Windows (x86_64), the package downloads a precompiled NIF and needs no compiler. Elsewhere, such as
on Alpine Linux or FreeBSD, it compiles the NIF, which needs a C compiler and make. See
Building from source.
{:ok, "é"} = Mojibake.normalize("é")
{:ok, "İSTANBUL"} = Mojibake.upcase("istanbul", locale: "tr")
true = Mojibake.caseless_equal?("Straße", "STRASSE")
["Hello", ",", " ", "world", "!"] = Mojibake.words("Hello, world!")
["Hello. ", "How are you?"] = Mojibake.sentences("Hello. How are you?")
[allowed: 6, mandatory: 11] = Mojibake.line_breaks("Hello world")
6 = Mojibake.terminal_width("日本語")
"日本" = Mojibake.truncate_to_width("日本語", 5)
"GRINNING FACE" = Mojibake.Char.name(0x1F600)
:greek = Mojibake.Char.script(?Ω)
{1, 2} = Mojibake.Char.numeric_value(?½)
true = Mojibake.Char.property?(?\s, :white_space)
%{type: :flag} = Mojibake.Emoji.sequence("🇮🇹")
["apple", "Apple", "éclair", "zebra"] = Mojibake.Collation.sort(["zebra", "éclair", "apple", "Apple"])
{:ok, "xn--bcher-kva.example"} = Mojibake.IDNA.to_ascii("Bücher.example")
true = Mojibake.Security.confusable?("paypal", "pаypal")
"sr-Latn-RS" = to_string(Mojibake.LanguageTag.parse!("SR-latn-rs"))
"abc גבא" = Mojibake.Bidi.visual("abc אבג")
What it adds to String
String has UTF-8 strings, graphemes, the four normalization forms, and upper and lowercase with
:turkic and :greek modes. Its Unicode data is the one built into your Elixir release. This
package adds:
| Area | API |
|---|---|
| Normalization (UAX #15) | normalize/3 with :nfkc_casefold, normalized?/2, quick_check/2 |
| Casing with any BCP 47 locale | upcase/2, downcase/2, titlecase/2 with locale: "lt" |
| Case folding | casefold/2, full or simple |
| Caseless matching (D144-D147) | caseless_equal?/3 |
| Word and sentence segmentation (UAX #29) | words/1, sentences/1, word_count/1, sentence_count/1, word_prefix/2 |
| Line breaking (UAX #14) | line_breaks/1 |
| Graphemes (UAX #29) | graphemes/1, grapheme_count/1, grapheme_prefix/2 |
| Terminal width (UAX #11) | terminal_width/2, truncate_to_width/3 |
| Malformed UTF-8 | a :malformed option on every transform, and errors with the byte offset |
| Character data | Mojibake.Char: names, categories, combining classes, bidi classes, decomposition types, numeric values, simple case mappings, blocks, scripts and Script_Extensions, East Asian width, emoji properties, and every stored binary and enumerated property |
| Scripts, blocks, planes, properties | Mojibake.Script, Mojibake.Block, Mojibake.Plane, Mojibake.Property |
| Emoji sequences (UTS #51) | Mojibake.Emoji.sequence/1, rgi?/1 |
| Hangul | Mojibake.Hangul: composition, decomposition, syllable names |
| Collation (UTS #10) | Mojibake.Collation: compare/3, sort/2, sort_key/2, and an Enum.sort/2 sorter |
| Internationalized domain names (UTS #46) | Mojibake.IDNA: to_ascii/1, to_unicode/1 |
| Confusables and mixed scripts (UTS #39) | Mojibake.Security: confusable?/2, skeleton/1, resolved_scripts/1 |
| Identifiers (UAX #31) | Mojibake.Identifier: valid?/2, start?/2, continue?/2 |
| Encodings | Mojibake.Encoding: UTF-8, UTF-16, and UTF-32 detection, validation, and conversion with byte offsets in errors |
| BCP 47 language tags | Mojibake.LanguageTag: strict parsing and formatting |
| Bidirectional text (UAX #9) | Mojibake.Bidi: embedding levels, visual reordering, level runs, and mirroring |
Design
- Transforms return
{:ok, string}or{:error, %Mojibake.Error{}}, and have a!variant that raises. When the input is already in the requested form, the input binary itself is returned, without a copy. - Queries such as
graphemes/1,grapheme_count/1, andterminal_width/2never fail: they treat each maximal ill-formed subsequence as U+FFFD. - Segments are sub-binaries of the input, so splitting a string does not copy it.
- A resolved bidi paragraph is a NIF resource: it is resolved once, can be shared between processes, and is freed when it is garbage collected.
- Casing takes the locale as an option, so the package never touches the C library's process-global casing locale.
- Inputs larger than 16 KiB run on a dirty CPU scheduler, so long strings do not block the normal schedulers.
- Character data uses atoms made from the long UCD names, such as
:uppercase_letter,:latin,:basic_latin, and:white_space. The atom tables are generated from the same Unicode data as the C library, so they always agree with it. - Codepoint functions take integers (
?A,0x1F600) and raiseFunctionClauseErrorfor anything outside 0 to 0x10FFFF. - The C library allocates through
enif_alloc, so its memory shows up in the VM's allocator statistics.
Optional features
The C library has four optional features: :names (character names), :collation, :idna, and
:security. They are all on by default, and Mojibake.features/0 lists the ones compiled in.
Mojibake.Char.name/1 needs :names, Mojibake.Collation needs :collation, Mojibake.IDNA
needs :idna, and Mojibake.Security needs :security; without them they return or raise
Mojibake.Error with reason :feature_not_enabled. Turn off the ones you do not need to make the
NIF smaller:
# config/config.exs
config :mojibake, features: []
Run mix deps.compile mojibake --force after changing it. The MOJIBAKE_FEATURES environment
variable, a comma-separated list or none, wins over the configuration. The precompiled NIFs have
every feature, so a package with fewer features compiles from source.
Building from source
The package compiles the NIF instead of downloading it when:
- there is no precompiled NIF for the platform, or downloading it fails
config :mojibake, features: [...]orMOJIBAKE_FEATURESleaves out a feature- the
MOJIBAKE_BUILDenvironment variable is1ortrue, or the configuration hasconfig :elixir_make, :force_build, mojibake: true
Compiling needs a C11 compiler and make: on Debian and Ubuntu the build-essential package, on
Alpine build-base, and on macOS the Xcode command-line tools. On Windows it needs Visual Studio
2019 16.8 or newer with the C++ tools: run mix from an x64 Native Tools Command Prompt, so that
cl and nmake are on the PATH. The C sources ship with the package; no CMake or system library
is needed.
The precompiled NIFs are assets of the GitHub releases
of Mojibake, checked against the checksums in the package. To download them from a mirror, set
MOJIBAKE_PRECOMPILED_URL to a URL template with @{artefact_filename} for the file name, such as
https://mirror.example/mojibake/@{artefact_filename}.
Contributing
The package lives in the Mojibake repository, in
src/elixir/. Inside the repository it compiles the C sources in src/; the Hex package carries
a copy. From the repository root:
make test-elixirruns the formatter check, the tests with all features on and then off, and the docs build, all with warnings as errors. The conformance tests read the Unicode test files that./utils/generate/scripts/download-test-data.shdownloads; they are skipped if those are missing.make package-elixircopies the C sources into the package, builds it withmix hex.build, and checks that the unpacked package compiles and loads. Nothing is published.
License
MIT. See LICENSE.