Iconvex Standards

iconvex_standards adds the standards, registry, and archival codecs that do not belong in the small Iconvex core. It is pure Elixir: no NIF, port, or system iconv process is used.

Starting the OTP application atomically installs 248 collision-reviewed byte codecs and 205 table providers into Iconvex. Stopping it removes exactly that extension. A failed or conflicting installation is rolled back in full.

Install

def deps do
  [
    {:iconvex, "~> 0.1"},
    {:iconvex_standards, "~> 0.1"}
  ]
end

The application starts automatically:

{:ok, koi8} = Iconvex.convert("Привет", "UTF-8", "KOI8-F")
{:ok, "Привет"} = Iconvex.convert(koi8, "KOI8-F", "UTF-8")

{:ok, bootstring} =
  Iconvex.convert("Καλημέρα 日本語", "UTF-8", "PUNYCODE")

{:ok, "Καλημέρα 日本語"} =
  Iconvex.convert(bootstring, "PUNYCODE", "UTF-8")

{:ok, mailbox_name} =
  Iconvex.convert("旅行/日本", "UTF-8", "UTF-7-IMAP")

All normal Iconvex conversion policies remain available, including strict errors, discard, byte substitution, and Unicode substitution.

What is in the package?

The frozen package manifest contains:

Surface	Count	Highlights
Registered byte codecs	248	143 RFC 1345 sets, ISO-IR, ISCII, Punycode, IMAP UTF-7, VIQR, VSCII-2, PASCII, KOI, Kermit, TACE-16
Table providers	205	Package-local, immutable ETF lookup tables
Packed profiles	5	ECMA-1 six-bit plus four seven-bit profiles, each MSB- and LSB-first
Property-token mappings	4	Unicode 17 mainland/Taiwan telegraph and kGB3
Raw transports	2	ECMA-44 seven- and eight-bit punched-card combinations
Catalogued quarantine	2	Fully mapped RFC 1345 tables intentionally not registered

The larger generated families are 143 registered RFC 1345 codecs, 14 IANA PCL symbol sets, 12 modern ISO-IR sets, seven CNS 11643 sets, six historical graphic sets, five mosaic/technical sets, four Unicode legacy mappings, three JIS X 0213 sets, and two KPS 9566-97 profiles.

Notable algorithmic and source-qualified codecs include:

PUNYCODE, UTF-7-IMAP, VIQR, MNEMONIC, and MNEM
all ten ISCII 1991 scripts
KOI7-switched, KOI8-F, draft KOI8-C, and SHORT-KOI
ABICOMP, BRASCII, VSCII-2, and four explicit PASCII policies
ECMA-1, ISO 10585, ISO-IR 42, ISO-IR 169, and fixed-width Japanese EUC
three Kermit ISO profiles and three commit-qualified Lithuanian standards
TACE-16 in both big- and little-endian word transports

Iconvex.Standards.encodings/0 returns the exact registered canonical names. Iconvex.Standards.registrations/0 also exposes aliases and source provenance. The machine-readable ownership authority is SURFACE_MANIFEST.tsv.

Odd-width transports without pretending bits are bytes

The byte codec API represents one six- or seven-bit unit per octet. For wire formats, use the explicit packed facade:

alias Iconvex.Standards.Packed

{:ok, bits} =
  Packed.encode_from_utf8("ABC", "ECMA-1-PACKED-MSB")

{:ok, "ABC"} =
  Packed.decode_to_utf8(bits, "ECMA-1-PACKED-MSB")

{:ok, lsb_transport} =
  Packed.encode_from_utf8("Привет", "SHORT-KOI-PACKED-LSB")

{:ok, "Привет"} =
  Packed.decode_to_utf8(lsb_transport, "SHORT-KOI-PACKED-LSB")

LSB-first data is returned as Iconvex.Packed.LSB, retaining the exact bit length, unit width, and bit order. Passing an explicitly named MSB profile with :lsb (or vice versa) is rejected instead of silently changing the transport.

Unicode property tokens are not byte encodings

The Unihan telegraph and kGB3 values are single property tokens. They therefore implement Iconvex.Standards.PropertyTokenMapping, not Iconvex.Codec:

alias Iconvex.Standards.Unihan17MainlandTelegraphDecimalToken, as: Telegraph

{:ok, scalar} = Telegraph.decode_token("0001")
{:ok, token} = Telegraph.encode_scalar(scalar)

Taiwan telegraph offers separate readable and lossless-VPUA reverse policies. The distinction prevents duplicate and normalization-sensitive property values from being presented as a fictitious concatenated byte-stream encoding.

Verification

The package's RED/GREEN suite includes the 197 migrated source tests plus package-boundary tests. It exhaustively traverses every decode and canonical encode entry in all 205 provider tables, all single-byte inputs, all 13,069 ISCII oracle vectors, every TACE-16 word in both byte orders, every packed source unit in both bit orders, and all 7,236 kGB3 property/GL rows. Lifecycle tests force a provider collision and verify complete rollback.

Run it locally:

ICONVEX_PATH=../iconvex mix test --warnings-as-errors
elixir tools/generate_surface_module.exs --check

Performance and scaling gates live in bench/ and are documented in BENCHMARKS.md. Test chronology is retained in TDD_LOG.md.

License and provenance

Original Iconvex code is LGPL-2.1-or-later, matching GNU libiconv. Packaged upstream evidence retains its own Unicode, BSD-3-Clause, MIT, or PSF terms. See NOTICE and the accompanying license files.