Iconvex
2,105 codecs. Ten runtime packages. Pure Elixir/BEAM. Zero NIFs, ports,
operating-system iconv, or external processes.
Iconvex begins as a native BEAM port of GNU libiconv 1.19, including every
codec selected by --enable-extra-encodings. Then it wanders happily into the
rest of character-set history: GSM septets and national shift tables, Morse
and Baudot, UTF-9 nonets, UTF-18 words, Punycode, SCSU, MARC-8, PETSCII,
ATASCII, DEC SIXBIT and RADIX-50, UNIVAC FIELDATA, punched cards, TeX fonts,
vendor compatibility pages, current ICU converters, and 1,050 pinned
historical ICU revisions.
Yes: 1,050 historical ICU revisions, each source-filename-qualified instead of silently collapsing incompatible versions behind one friendly alias.
GNU libiconv 1.19 fixed-codec parity remains 198/198: 112 default codecs in Core and 86 optional codecs in Extras. The 2,105 unique canonical codecs number is the larger Iconvex ecosystem, not a redefinition of GNU's surface.
The ordinary API stays ordinary:
{:ok, "café"} =
Iconvex.convert(<<0x63, 0x61, 0x66, 0xE9>>, "CP1252", "UTF-8")
When eight-bit binaries would be a lie, package APIs return exact Elixir bitstrings or self-describing LSB containers.
Pick your package
| Package | Canonical codecs | The rabbit hole |
|---|---|---|
iconvex | 112 | GNU default: Unicode, CJK, ISO-2022, HZ, Windows/Mac/ISO pages, C99/JAVA, transliteration, generic 1–8-bit packing |
iconvex_extras | 86 | Complete GNU --enable-extra-encodings: AIX, DOS, OSF/1, z/OS EBCDIC, Atari ST, RISC OS, DEC Asian, JIS X 0213 |
iconvex_telecom | 61 | GSM 03.38 national tables, Morse, Baudot/ITA, SITOR/NAVTEX, AIS6, TBCD, SIM alphabets, broadcast and teleprinter formats |
iconvex_unicode | 17 | BOCU-1, CESU-8, SCSU, UTF-EBCDIC, UTF-1/5/6, WTF-8, signature profiles, plus native UTF-9 and UTF-18 APIs |
iconvex_standards | 248 | RFC 1345, Punycode, IMAP UTF-7, ISCII, ISO-IR, ECMA, PCL, Unihan and standards-defined transforms |
iconvex_bibliographic | 10 | MARC-8, ANSEL, Cork/T1, TeX OT1/OML/OMS, PostScript and glyph-vector mappings |
iconvex_retro | 192 | PETSCII, ATASCII, Apple/DEC/CDC/UNIVAC, terminals, calculators, punched cards and 57 packed profiles |
iconvex_platforms | 117 | IBM, Microsoft, .NET, Java, CPython, glibc, Kermit, Adobe, Mac and other vendor-defined compatibility surfaces |
iconvex_icu | 212 | Current ICU-derived SBCS/MBCS/stateful tables, LMBCS variants, Unicode profiles and compound text |
iconvex_icu_archive | 1,050 | Source-filename-qualified historical ICU revisions, backed by six byte-balanced data packages |
Every package is independently installable and depends only on Core, except the Archive facade, which pulls its six exact-version provider-only data packages. Starting a leaf atomically registers its complete codec set. Install only the families the application actually uses.
iconvex_specs is an unpublished generation, research, conformance, and
integration workspace. It owns no runtime codec and must not be an application
dependency.
For the whole map, open the generated
codec-to-package index
or its machine-readable
ICONVEX_CODEC_PACKAGE_INDEX.csv.
The full GNU comparison
lists all 2,105 names with ownership and GNU status. The
package architecture
documents dependencies, lifecycle, artifacts, and release order.
Installation
Core only:
def deps do
[{:iconvex, "~> 0.1"}]
end
Exact GNU fixed-codec surface:
def deps do
[
{:iconvex, "~> 0.1"},
{:iconvex_extras, "~> 0.1"}
]
end
Choose any leaves:
def deps do
[
{:iconvex, "~> 0.1"},
{:iconvex_telecom, "~> 0.1"},
{:iconvex_unicode, "~> 0.1"},
{:iconvex_standards, "~> 0.1"},
{:iconvex_bibliographic, "~> 0.1"},
{:iconvex_retro, "~> 0.1"},
{:iconvex_platforms, "~> 0.1"},
{:iconvex_icu, "~> 0.1"}
]
end
For the historical archive, add only the facade:
{:iconvex_icu_archive, "~> 0.1"}
Mix fetches iconvex_icu_archive_data_01 through
iconvex_icu_archive_data_06 transitively. Those packages own table storage,
not additional codec names.
Encoding gallery
These examples are executable integration tests, not aspirational snippets.
Core: escapes, state machines, and Unicode
Core owns exactly GNU's 112 default codecs and accepts 446 normalized names. C99/JAVA source escapes are codecs, and HZ remains a state machine:
{:ok, ~S(\u00e9\U0001f600)} =
Iconvex.convert("é😀", "UTF-8", "C99")
{:ok, "~{VP~}~~{ND~}"} =
Iconvex.convert("中~文", "UTF-8", "HZ")
[<<"caf", 0xC3>>, <<0xA9>>]
|> Iconvex.stream!("UTF-8", "CP1252")
|> Enum.join()
#=> <<0x63, 0x61, 0x66, 0xE9>>
Highlights include UTF-7/8/16/32, UCS-2/4, GB18030:2005 and :2022, EUC-JP/KR/TW, Big5/HKSCS, Shift_JIS, JOHAB, TCVN, VISCII, and the complete ISO-2022-JP/JP-1/JP-2/JP-MS/KR/CN/CN-EXT family.
Extras: GNU's whole weird attic
Extras is the exact 86-codec complement behind GNU's optional definition
families. It includes DOS pages, AIX, OSF/1, 53 z/OS pages, ATARIST,
RISCOS-LATIN1, DEC-HANYU, BIG5-2003, JIS X 0213 and
ISO-2022-JP-3.
The z/OS UNIX surface makes EBCDIC newline policy explicit:
{:ok, ebcdic} =
Iconvex.convert("hello\n", "ASCII", "IBM-1047/ZOS_UNIX")
<<0x88, 0x85, 0x93, 0x93, 0x96, 0x15>> = ebcdic
{:ok, "hello\n"} = Iconvex.convert(ebcdic, "IBM-1047/ZOS_UNIX", "ASCII")
IBM-423 is here too: the Greek EBCDIC codec grounded in the original glibc implementation.
Telecom: septets, Baudot, Morse, and radio alphabets
Telecom owns 61 byte codecs and 53 packed profiles spanning 3, 5, 6, and 7 bits. International Morse follows ITU-R M.1677:
{:ok, "... --- ..."} = Iconvex.convert("SOS", "UTF-8", "MORSE-CODE")
{:ok, "SOS"} = Iconvex.convert("... --- ...", "MORSE-CODE", "UTF-8")
# MORSE-CODE resolves to MORSE-ITU-M1677.
GSM 03.38's registry surface uses one septet per octet, exactly as SMPP does. The high bit stays clear:
{:ok, <<0x04, 0x0B>>} = Iconvex.convert("€Ğ", "UTF-8", "GSM0338-TURKISH")
{:ok, "€Ğ"} = Iconvex.convert(<<0x04, 0x0B>>, "GSM0338-TURKISH", "UTF-8")
All national locking and single-shift tables are available. They can also be paired independently:
alias Iconvex.Telecom.GSM0338
options = [locking: :portuguese, single_shift: :bengali]
{:ok, [0x00EA, 0x09E6]} = GSM0338.decode(<<0x04, 0x1B, 0x19>>, options)
{:ok, <<0x04, 0x1B, 0x19>>} = GSM0338.encode([0x00EA, 0x09E6], options)
182 = length(GSM0338.valid_pairs())
TPDU packing produces the actual contiguous wire septets:
alias Iconvex.Telecom.GSM0338.Packing
{:ok, packed} = Packing.pack("hellohello")
"E8329BFD4697D9EC37" = Base.encode16(packed)
{:ok, "hellohello"} = Packing.unpack(packed, 10)
The same package covers ITA1/ITA2, Murray 1905, Wheatstone/Hausmann, CCIR476/SITOR/NAVTEX, AIS6, TBCD, SIM/USIM alphabets, ARIB and three Unicode 17 carrier-emoji profiles.
Unicode: nonets, 18-bit words, and compression
Unicode owns BOCU-1, CESU-8, FSS-UTF, SCSU, UTF-EBCDIC, UTF-1, UTF-5, UTF-6, WTF-8 and explicit signature profiles. UTF-9 and UTF-18 are logical non-octet codecs, so their native APIs preserve the real width:
{:ok, utf9} = Iconvex.Unicode.UTF9.encode_packed([?A, 0x0391, 0x10330])
54 = bit_size(utf9)
{:ok, [?A, 0x0391, 0x10330]} = Iconvex.Unicode.UTF9.decode_packed(utf9)
{:ok, utf18} = Iconvex.Unicode.UTF18.encode_packed([?A, 0x10330])
36 = bit_size(utf18)
{:ok, [?A, 0x10330]} = Iconvex.Unicode.UTF18.decode_packed(utf18)
Byte protocols use UTF-9-16BE, UTF-9-16LE, UTF-18-24BE, or
UTF-18-24LE. Padding and endianness are part of the name instead of ambient
machine state.
Standards: RFCs, ISO registers, and raw Punycode
Punycode is the raw RFC 3492 transform; it deliberately does not invent an
xn-- DNS prefix:
{:ok, "ihqwcrb4cv8a8dqg056pqjye"} =
Iconvex.convert("他们为什么不说中文", "UTF-8", "PUNYCODE")
{:ok, "他们为什么不说中文"} =
Iconvex.convert("ihqwcrb4cv8a8dqg056pqjye", "PUNYCODE", "UTF-8")
This is also where RFC 1345 identities, IMAP UTF-7, ISCII, KOI variants,
ECMA-1/44, PCL symbol sets, ISO-IR registers and property-token mappings live.
GNU spellings that collide with historical RFC 1345 names keep the GNU route;
the historical identity remains explicit as RFC1345:IBM037, and so on.
Bibliographic: MARC-8, ANSEL, and TeX machinery
MARC-8 puts combining marks before the base letter. Unicode does the opposite. The codec knows:
{:ok, "e\u0301"} = Iconvex.convert(<<0xE2, ?e>>, "MARC-8", "UTF-8")
The package also owns ANSEL, Cork/T1, OT1, OML, OMS, Adobe PostScript and AGL glyph-vector mappings, including explicit packed seven-bit math-font profiles.
Retro: six-bit machines, punched cards, and calculator glyphs
The Apple-1 Signetics 2513 address order is @–_, then space–?. The
registry exposes one unit per byte; Iconvex.Retro.Packed preserves six bits:
{:ok, <<0, 1, 2, 3, 32, 63>>} =
Iconvex.convert("@ABC ?", "UTF-8", "APPLE-1-SIGNETICS-2513-CM2140-6BIT")
{:ok, packed} =
Iconvex.Retro.Packed.encode_from_utf8(
"@ABC ?",
Iconvex.Retro.Apple1Signetics2513,
:msb
)
36 = bit_size(packed)
{:ok, "@ABC ?"} =
Iconvex.Retro.Packed.decode_to_utf8(
packed,
Iconvex.Retro.Apple1Signetics2513,
:msb
)
DEC RADIX-50 really is a base-40 word:
{:ok, <<0o115402::16-little>>} =
Iconvex.convert("X2B", "UTF-8", "DEC-RADIX-50")
Other delights include PETSCII, ATASCII, Apple II, ZX80/81/Spectrum,
Amstrad, MSX, TRS-80, TI calculators, DEC-SIXBIT,
FIELDATA-UNIVAC-1100, CDC display codes, Hollerith and 12-bit punched cards.
Platforms: vendor reality is also a specification
Platforms preserves source-qualified vendor behavior instead of flattening near-matches into a convenient fiction:
{:ok, "\u1E83"} = Iconvex.convert(<<0xFF>>, "MacCeltic", "UTF-8")
IBM-423 and GNU's optional IBM pages remain in Extras; additional IBM CCSIDs, Windows best-fit maps, Java modified UTF-8, .NET/CPython profiles, glibc charmaps, Adobe mappings and Mac variants live here.
ICU: current converter profiles
Current ICU data and algorithm profiles are separate from historical revisions. LMBCS optimization groups remain distinct codecs:
{:ok, <<0xD5>>} = Iconvex.convert("Ω", "UTF-8", "LMBCS-2")
{:ok, "Ω"} = Iconvex.convert(<<0xD5>>, "LMBCS-2", "UTF-8")
The 212-codec package includes 191 generated table providers plus compound text, JIS, LMBCS and Unicode variant implementations.
ICU Archive: reproducible historical behavior
The Archive owns 1,050 source-filename-qualified revisions. Six provider-only packages keep artifacts manageable without adding duplicate codec identities. Here is a stateful IBM-930 revision, including SO/SI:
codec = Iconvex.ICUArchive.Codecs.C674
bytes = <<0xC1, 0x0E, 0x44, 0x81, 0x44, 0x82, 0x0F, 0xC2>>
"ICU-ARCHIVE-ibm-930_P120-1999" = codec.canonical_name()
{:ok, ^bytes} = codec.encode(~c"AあいB")
{:ok, ~c"AあいB"} = codec.decode(bytes)
The facade manifest, shard assignment, every table digest and every generated decoder/encoder mapping are exhaustively checked.
Daily conversion API
Bang functions return the binary directly. Suffixes and explicit options share one implementation:
<<"caf", 0xE9>> = Iconvex.convert!("café", "UTF-8", "CP1252")
{:ok, "AB"} = Iconvex.convert("A😀B", "UTF-8", "ASCII//NON_IDENTICAL_DISCARD")
Options can also be passed explicitly:
Iconvex.convert(input, from, to,
invalid: :discard,
unrepresentable: :discard,
transliterate: true,
byte_substitute: "<0x%02x>",
unicode_substitute: "<U+%04X>",
on_invalid_byte: fn event -> {:replace, "<byte-#{event.byte}>"} end
)
The byte_substitute option replaces malformed source bytes; unicode_substitute
replaces characters unavailable in the destination. The formats accept GNU-style
%x/%X fields, including zero-padded widths. With iconvex_extras installed,
IBM-1047 newline behavior can be selected by name or explicitly:
Default destination handling mirrors GNU libiconv. An ordinary character that
the target cannot represent is an error. Extended UCS values that reach a
Unicode-family target use U+FFFD when that target defines and can encode the
fallback; unrepresentable: :discard, transliteration, or
unicode_substitute takes precedence. JAVA and UTF-7 intentionally preserve
isolated 16-bit surrogate units when the source codec exposes them. Generic
UTF-16 and UTF-32 emit a BOM only when at least one target character is emitted.
ISO-2022-JP-2 also implements GNU's case-insensitive ja, ko, and zh
Unicode language tags, including cancellation and streaming state.
Iconvex.convert(input, "IBM-1047/ZOS_UNIX", "UTF-8")
Iconvex.convert(input, "UTF-8", "IBM-1047", to_surface: :zos_unix)
on_invalid_byte handles source bytes a codec cannot decode. The callback gets
an Iconvex.InvalidByte containing canonical encoding, error kind, absolute
offset, current byte, and codec error sequence. It returns :default, :error,
:discard, or {:replace, utf8_or_codepoints}. Iconvex does not label an
unmapped byte as a display command, corruption, or application control. That
decision belongs to the file format and caller; the same callback works in
one-shot and lazy Stream conversion.
External fixed-width codecs can declare multi-byte recovery consumption, and whole-string transforms can declare stop-on-error recovery so callbacks never restart a desynchronized suffix. See EXTENDING.md.
Errors contain the source byte offset or unrepresentable Unicode code point:
{:error, %Iconvex.Error{kind: :invalid_sequence, offset: 3}} =
Iconvex.convert(input, "UTF-8", "CP1252")
For lazy bounded-memory conversion, input enumerable emits binaries and output is an Elixir Stream of converted binaries:
output =
File.stream!(path, [], 64 * 1024)
|> Iconvex.stream!("PETSCII", "UTF-8", on_invalid_byte: controls)
|> Enum.into("")
Multibyte lookahead, multi-code-point target mappings, source offsets, UTF BOM
state, and built-in HZ/ISO-2022/UTF-7 state survive chunk boundaries. A UTF-7
shifted run is committed only after its base64 padding and complete UTF-16
payload have been validated; a plus inside an active shift remains in that
Base64 run, and byte-at-a-time shifted input is processed in linear time.
ISO-2022 diagnostic frames and JP-MS SO/SI state are likewise split-invariant.
Output chunks need not align with input chunks. Conversion failures raise
Iconvex.Error during enumeration because earlier output may already have
been consumed. External codecs use the incremental callbacks documented in
EXTENDING.md.
For a buffered chunked transaction:
{:ok, converter} = Iconvex.new("UTF-8", "SHIFT_JIS")
{:ok, <<>>, converter} = Iconvex.feed(converter, chunk1)
{:ok, <<>>, converter} = Iconvex.feed(converter, chunk2)
{:ok, output, finished} = Iconvex.finish_with_state(converter)
{:error, :already_finished} = Iconvex.feed(finished, chunk3)
Every source is buffered until finish/1 or finish_with_state/1. Each feed is
O(1), and finalization performs one conversion over the accumulated iodata.
This contract makes chunked output byte-identical to one-shot output across
every split, including stateful designations, malformed-input policies,
Vietnamese composition, multi-code-point destination mappings, and UTF BOMs.
Errors use offsets from the beginning of the complete stream. finish/1 keeps
the original compatibility return shape; use finish_with_state/1 when the
terminal converter state and repeated-finish rejection are required.
Use Iconvex.encodings/0 for canonical names and
Iconvex.canonical_name/1 to resolve an alias.
Non-octet packed transports
Codecs that expose one complete signal unit per octet can be packed without copying their mapping logic:
{:ok, packed} = Iconvex.Packed.encode_from_utf8("Hello", "ASCII", 7)
{:ok, "Hello"} = Iconvex.Packed.decode_to_utf8(packed, "ASCII", 7)
{:ok, lsb} = Iconvex.Packed.encode_from_utf8_lsb("hellohello", "ASCII", 7)
"E8329BFD4697D9EC37" = Base.encode16(lsb.data)
{:ok, "hellohello"} = Iconvex.Packed.decode_to_utf8_lsb(lsb, "ASCII")
pack/2 returns an exact MSB-first Elixir bitstring. pack_lsb/2 returns an
Iconvex.Packed.LSB value carrying the byte data, exact meaningful bit count,
and unit width; this prevents final-octet padding from becoming ambiguous.
If unpacking succeeds but the named codec rejects one or more logical units,
the returned Iconvex.Error uses offset_unit: :bit and a physical bit
offset. MSB errors retain the exact offending bitstring; LSB errors retain a
self-describing Iconvex.Packed.LSB fragment rather than exposing temporary
one-byte-per-unit coordinates.
Every unit value and width 1–8 is exhaustively tested in both orders. Wider
formats with codec-specific scalar rules live in their owning leaves:
iconvex_unicode owns UTF-9/UTF-18, while Standards, Bibliographic, Retro, and
Telecom publish their own exact profile catalogs.
External codec libraries
External packages implement Iconvex.Codec, then atomically register a module
at startup:
defmodule MyLibrary.Codec do
use Iconvex.Codec
def canonical_name, do: "X-MY-CODEC"
def decode(binary), do: MyLibrary.Decoder.decode(binary)
def decode_discard(binary), do: MyLibrary.Decoder.decode_discard(binary)
def encode(codepoints), do: MyLibrary.Encoder.encode(codepoints)
def encode_discard(codepoints), do: MyLibrary.Encoder.encode_discard(codepoints)
def encode_substitute(codepoints, replacer),
do: MyLibrary.Encoder.encode_substitute(codepoints, replacer)
end
{:ok, ownership} = Iconvex.register_codec_if_absent(MyLibrary.Codec)
{:ok, output} = Iconvex.convert(input, "X-MY-CODEC", "UTF-8")
If ownership is a reference, retain it and conditionally unregister with
Iconvex.unregister_codec(MyLibrary.Codec, ownership) on shutdown. The
:existing result means another caller owns the registration and it must be
left in place. Versioned recovery snapshots preserve registrations and exact
ownership tokens across consecutive supervised registry-worker crashes while
the :iconvex application remains running. A clean application stop erases
that recovery state, so registrations never cross a deliberate stop/start
boundary. Replacement publishes at one commit point, so concurrent lookups
resolve the complete old registration or the complete new registration.
Strict per-commit durability is deliberate. Every successful registry
mutation writes a synchronous recovery snapshot of the complete registry to
:persistent_term; replacing that term can schedule VM-wide garbage
collection. Keep registration on the control plane rather than a request hot
path. Package authors with many codecs should use
Iconvex.Extension.register/2, which wraps the internal register_set
transaction, publishes the complete set atomically, and amortizes the snapshot
to one checkpoint. Debouncing this write would weaken the documented
consecutive worker-crash guarantee.
Mapping tables and their dense/sparse decode accelerators are also lazy
:persistent_term entries. The shipped resident set is bounded by the finite
codec inventories of the applications actually loaded; each table identity
has one base term and at most one term for each of the three derived cache
shapes. Unregistering a table provider erases its base term and all three
derived keys under their build locks, so repeated external-package
load/use/unload cycles do not accumulate provider cache generations. Provider
startup and shutdown therefore belong on the control plane as well; arbitrary
hot-path LRU replacement would impose the VM-wide cost that these immutable
caches are designed to avoid.
If a module's declared name collides with another source, register it under an explicit source-qualified name. The declared canonical name is replaced for that registration; module aliases and option aliases are still registered and collision-checked:
:ok =
Iconvex.register_codec(MyLibrary.Codec,
canonical: "MY-SOURCE-X-MY-CODEC",
aliases: ["MY-SOURCE-MYCODEC"]
)
Consumer configuration can load modules automatically with
config :iconvex, external_codecs: [MyLibrary.Codec]. Tuple entries accept the
same :canonical and :aliases options. Registered codecs appear
in Iconvex.encodings/0; built-in names cannot be overridden.
External OTP applications that own a complete codec inventory and mapping tables can publish both with one public package transaction:
{:ok, token} =
Iconvex.Extension.register(
[MyLibrary.Codec, MyLibrary.OtherCodec],
owner: :my_library,
priority: 10,
table_providers: [:my_mapping]
)
# Keep the opaque token in Application state.
:ok = Iconvex.Extension.unregister(token)
Registration and removal are atomic to concurrent lookups, failures roll back
newly acquired providers, supervised worker restarts preserve the exact token,
and stale tokens cannot remove replacements. See
EXTENDING.md for callback/error contracts, aliases, stateful
streaming, package priorities and collisions, required linear substitution,
optional zero-copy UTF-8 callbacks, performance results, and an external-library
test checklist. iconvex_extras is the reference external codec package: it
shares core conversion engines while keeping every optional mapping table and
codec module outside the core artifact.
Each of the nine extension facades installs one atomic managed registration
set. The integration contract exercises forward, reverse, and every cyclic
start boundary and requires the same 2,105-name registry. Canonical claims rank
ahead of aliases, then by fixed package priority; stopping a winner exposes a
retained fallback without a transient lookup gap. Individual third-party
registrations through register_codec* remain strict: any distinct module
claiming an existing name gets a {:name_conflict, name} error rather than
silently joining a managed set.
The generated full-stack codec comparison records the owning package and GNU libiconv status for every canonical name. The richer codec-to-package index adds modules, aliases, sources, surface kinds, packed profiles, provenance records, and install snippets.
Exhaustive means exhaustive
Counts in this README come from checked-in generated inventories, not hand-kept marketing lists:
- Core registers 112/112 GNU-default codecs.
- Core + Extras register all 198 GNU libiconv 1.19 codecs, including every
codec behind
--enable-extra-encodings. - Telecom registers 61 codecs and publishes 53 fixed-width packed profiles.
- Unicode registers 17 byte codecs and exposes two additional logical non-octet APIs.
- Standards, Bibliographic, Retro, Platforms, and current ICU register 248 + 10 + 192 + 117 + 212 codecs.
- The ICU Archive facade registers 1,050 historical revisions; its six provider packages add storage, not codec identities.
- The ten runtime packages expose exactly 2,105 unique canonical names.
The GNU differential walks one sequential UCS-4BE corpus containing all 1,114,112 code points from U+0000 through U+10FFFF—including scalar values and all 2,048 surrogate code points—in forward, reverse, and both cross-decoder directions. The combined performance gate takes the fastest of three isolated samples and rejects any supported GNU codec slower than 30x GNU libiconv.
Every leaf adds source-pinned vectors, full encode/decode table walks where applicable, malformed-input policies, every-split streaming checks, packed bit-order round trips, lifecycle tests, and clean artifact contracts. The README gallery itself is executable integration-test input.
The current research projection tracks 385 direct Wikipedia-sourced clusters absent from GNU libiconv 1.19. Of those, 249 are implemented by Iconvex. The remaining 136 are audited instead of guessed: 12 are actionable codec gaps, 96 still require exact public mapping/specification work, and 28 are families, mapping notations, repertoire abstractions/profiles, or a withdrawn unassigned part rather than standalone byte codecs. See the generated disposition report.
Operational and trust boundaries
- One-shot conversion and the buffered
new/feed/finishAPI materialize the complete source and, on general paths, a Unicode code-point list. Usestream/4orstream!/4for lazy bounded-memory conversion. Applications using the materializing APIs must enforce an input-size limit derived from their own memory budget; Iconvex deliberately does not choose a universal cap. - Runtime mapping tables load lazily. A process using every core table should
budget more than the measured 62.9 MiB flat decoded-table size, plus BEAM map,
code, output, and temporary-list overhead. Typical workloads load only touched
tables; optional mappings remain in
iconvex_extras/other external packages. - Substitution printf widths are validated and capped at 65,536 digits. Unknown, duplicate, or malformed options return typed request errors.
- Shipped table/transliteration ETF assets use safe decoding, versioned caches,
and serialized cold loads. External codec modules remain trusted application
code: callback exceptions may propagate. A malformed
encode_substitute/2result returns a typed{:invalid_codec_callback_return, ...}request error.
Verification
mix test
MIX_ENV=prod mix run bench/benchmark.exs
MIX_ENV=prod mix run bench/external_codec_benchmark.exs
GNU_ICONV=/path/to/gnu-libiconv-1.19/bin/iconv \
MIX_ENV=prod mix run tools/exhaustive_unicode_differential.exs
The public source monorepo is
edescourtis/iconvex. ExDoc source
links are pinned to the matching v0.1.1 tag and the iconvex/ subdirectory,
so published documentation resolves to the exact release source.
The repository carries
267 byte-exact upstream files plus one derived configured Makefile in its
268-file GNU libiconv 1.19 test-fixture set (9.0 MB). Separate
and combined aggregate SHA-256 values verify their provenance boundary. The set
is intentionally excluded from the consumer Hex artifact; generated runtime
tables and public conformance documents remain in that artifact. Core ExUnit
runs the applicable default-codec cases; iconvex_extras carries and runs the
92 extra charmap/inverse fixtures plus ISO-2022-JP-3 and IBM-1047 behavioral
cases.
Together the suites port the 164 stateless charmap/inverse checks, all 27 snippet
round-trips, the complete GB18030 supplementary range,
transliteration/substitution/EBCDIC/discard checks, and the four C regression
tests. The z/OS filesystem-tag check remains platform-only, just as upstream
skips it outside z/OS. See
UPSTREAM_TEST_COVERAGE.md for the machine-checked
traceability matrix. SUPPORTED_ENCODINGS.md lists all
198 codecs side-by-side with package ownership and GNU definition/default-build
status. Also see
EXHAUSTIVE_UNICODE_DIFFERENTIAL.md for the
112-codec core run, and the extras package's report for the combined 198-codec,
1,114,112-code-point forward/reverse/cross-decode differential against GNU
libiconv 1.19. The sequential UCS-4BE corpus contains all 1,112,064 Unicode
scalar values plus all 2,048 surrogate code points; surrogates are code points,
not scalar values. The combined directional performance gate records the
fastest of three isolated samples and enforces a 30x-GNU ceiling. Also see
BENCHMARKS.md and
TDD_LOG.md.
The disposition of every independent review finding is in
DEEP_DIVE_REMEDIATION.md.
elixir tools/release_check.exs builds docs, unpacks the exact Hex artifact into
a temporary directory, and verifies compilation and representative conversions
from a clean consumer project.
To regenerate the registry, mappings, extensions, and transliteration table from an unpacked GNU libiconv 1.19 source tree:
elixir tools/generate_registry.exs /path/to/libiconv-1.19
mix format
The generator writes the 112 default codecs into this package and the exact
86-codec complement into the sibling iconvex_extras package.
License
Iconvex and its derived GNU mapping/transliteration data are distributed under
LGPL-2.1-or-later. The repository-only upstream test mirror includes GNU's
test-only uniq-u.c, licensed GPL-3.0-only; that source and its license text are
excluded from the Hex artifact. See LICENSE,
GNU GPL 3.0, and NOTICE.