Getting Started with Cldr

Build StatusHex.pmHex.pmHex.pmHex.pm

Introduction

ex_cldr is an Elixir library for the Unicode Consortium’sCommon Locale Data Repository (CLDR). The intentions of CLDR, and this library, is to simplify the locale specific formatting and parsing of numbers, lists, currencies, calendars, units of measure and dates/times. As of April 8th 2021 and ex_cldr Version 2.20.0, ex_cldr is based upon CLDR version 39.0.

The first step is to define a module that will host the desired ex_cldr configuration and the functions that serve as the public API. This module is referred to in this documentation as a backend module. For example:

@doc """
Define a backend module that will host our
Cldr configuration and public API.

Most function calls in Cldr will be calls
to functions on this module.
"""
defmodule MyApp.Cldr do
  use Cldr,
    locales: ["en", "fr", "zh", "th"],
    default_locale: "en"

end

This strategy means that different configurations can be defined and it also means that one Cldr implementation won’t interfer with implementations in other, potentially dependent, applications.

The functions you are mostly likely to use are:

To access the raw Cldr data for a locale the Cldr.Config module is available. Note that the functions in Cldr.Config are typically used by library authors. The most useful function is:

Use Case

Use this library if you need to:

It is highly likely that you will also want to install one or more of the dependent packages that provide localization and formatting for a particular data domain. See Additional Cldr Packages below.

Elixir Version Requirements

Installation

Add ex_cldr and the JSON library of your choice as a dependencies to your mix project:

defp deps do
  [
    {:ex_cldr, "~> 2.23"},
    # Posion or any other compatible json library
    # that implements `encode!/1` and `decode!/1`
    # :jason is recommended
    {:jason, "~> 1.0"}
    # {:poison, "~> 2.1 or ~> 3.0"}
  ]
end

then retrieve ex_cldr and the JSON library from hex:

mix deps.get
mix deps.compile

Additional Cldr Packages

ex_cldr includes only basic functions to maintain the CLDR data repository in an accessible manner and to manage locale definitions. Additional functionality is available by adding additional packages:

Each of these packages includes ex_cldr as a dependency so configuring any of these additional packages will automatically install ex_cldr.

Configuration

Cldr attempts to maximise runtime performance at the expense of additional compile time. Where possible Cldr will create functions to encapsulate data at compile time. To perform these optimizations for all 541 locales known to Cldr wouldn’t be an effective use of your time or your computer’s. Therefore Cldr requires that you configure the locales you want to use.

The preferred way to configure Cldr is to define the configuration in your backend module. This removes any dependency on your mix.exs and therefore simplifies deployment as a release. However configuration can also be defined in other ways:

Global configuration.

In mix.exs a global configuration can be defined under the :ex_cldr key. Although any valid configuration keys can be used here, only the keys :json_library, :cacertfile and default_locale are considered valid. Other configuration keys may be used to aid migration from Cldr version 1.x but a deprecation message will be printed during compilation. Here’s an example of global configuration:

config :ex_cldr,
  default_locale: "en",
  default_backend: MyApp.Cldr,
  json_library: Jason,
  cacertfile: "path/to/cacertfile"

Note that the :json_library key can only be defined at the global level since it is required during compilation before any backend module is compiled.

On most platforms other than Windows the :cacertfile will be automatically detected. Any configured :cacertfile will take precedence on all platforms.

If configuration beyond the keys :default_locale, :cacertfile or :json_library are defined a deprecation warning is printed at compile time noting that configuration should be moved to a backend module.

Backend Module Configuration

The preferred configuration method is to define the configuration in the backend module. Using the backend configuration in config.exs is discouraged and will result in a warning at compile time. The configuration keys are the same so the preferred way to achieve the same configuration as defined in the global example is:

defmodule MyApp.Cldr do
  use Cldr,
    default_locale: "en",
    locales: ["fr", "en", "bs", "si", "ak", "th"],
    add_fallback_locales: false,
    gettext: MyApp.Gettext,
    data_dir: "./priv/cldr",
    otp_app: :my_app,
    precompile_number_formats: ["¤¤#,##0.##"],
    precompile_transliterations: [{:latn, :arab}, {:thai, :latn}],
    providers: [Cldr.Number],
    generate_docs: true,
    force_locale_download: false
end

Otp App Configuration

In the backend configuration example above the :otp_app key has been defined. This means that configuration for Cldr has been defined in mix.exs under the key :my_app with the sub-key MyApp.Cldr. For example:

defmodule MyApp.Cldr do
  use Cldr, otp_app: :my_app
end
# In mix.exs
config :my_app, MyApp.Cldr,
  default_locale: "en",
  locales: ["fr", "en", "bs", "si", "ak", "th"],
  gettext: MyApp.Gettext,
  data_dir: "./priv/cldr",
  precompile_number_formats: ["¤¤#,##0.##"],
  precompile_transliterations: [{:latn, :arab}, {:thai, :latn}]

Multiple backends can be configured under a single :otp_app if required.

Configuration Priority

When building the consolidated configuration the following priority applies:

Backend Configuration Keys

The configuration keys available for Cldr are:

use Cldr,
  default_locale: "en",
  locales: ["en-*", "fr"]
defmodule MyApp.Cldr do
  use Cldr,
    locales: ["en", "fr"],
    default_locale: "en",
    force_locale_download: Mix.env() == :prod
end

Providers

The data maintained by CLDR is quite large and not all capabilities are required by all applications. Hence Cldr has additional optional functionality that can be provided through additional hex packages. In order to support compile-time additions to a configured backend, any package can define a provider that will be called at compile time.

The currently known providers and their hex package names are:

Hex Package Provider Module Comment
ex_cldr_numbers Cldr.Number Formatting of numbers, currencies
ex_cldr_lists Cldr.List Formatting of lists
ex_cldr_units Cldr.Unit Formatting of SI and Imperial units
ex_cldr_currency Cldr.Currency Currency definitions and localizations
ex_cldr_territories Cldr.Territory Formatting of territory (country) data
ex_cldr_languages Cldr.Language Formatting of language information
ex_cldr_dates_times Cldr.DateTime Formatting of dates, times & datetimes
ex_cldr_locale_display Cldr.LocaleDisplay Localising locale names
ex_money Money Operations on and formatting of a money type
ex_messages Cldr.Message Formatting of ICU-formatted messages

Any library author can create a provider module by exposing a function called cldr_backend_provider/1 that takes a Cldr.Config struct as a single parameter. The function should return an AST that is inserted into the backend module being compiled.

Providers are configured on each backend module under the :providers key. It must be a list of provider modules. For example:

defmodule MyApp.Cldr do
  use Cldr,
    locales: ["en", "zh"],
    default_locale: "en",
    providers: [Cldr.Number, Cldr.List]
end

If :providers is nil (the default), Cldr will attempt to configure all of the providers described above if they have been installed as deps. If you don’t wish to invoke any providers, use the empty list [].

Migrating from Cldr 1.x

  1. Create a backend module by following the configuration instructions
  2. Delete any duplicated global configuration in any config.exs files. Only the keys :default_locale and :json_library are supported in the global configuration
  3. Update any plugs to configure the desired backend
  4. Adjust any API calls from Cldr.some_function to MyApp.Cldr.some_function. Or better still, alias your backend module where required. ie. alias MyApp.Cldr, as: Cldr

Downloading Locales

Cldr can be installed from either github or from hex.

Localizing Numbers

The Cldr.Number module implemented in the ex_cldr_numbers package provides number formatting. The public API for number formatting is MyApp.Cldr.Number.to_string/2. Some examples:

iex> MyApp.Cldr.Number.to_string 12345
"12,345"

iex> MyApp.Cldr.Number.to_string 12345, locale: "fr"
"12 345"

iex> MyApp.Cldr.Number.to_string 12345, locale: "fr", currency: "USD"
"12 345,00 $US"

iex> MyApp.Cldr.Number.to_string 12345, format: "#E0"
"1.2345E4"

iex(> MyApp.Cldr.Number.to_string 1234, format: :roman
"MCCXXXIV"

iex> MyApp.Cldr.Number.to_string 1234, format: :ordinal
"1,234th"

iex> MyApp.Cldr.Number.to_string 1234, format: :spellout
"one thousand two hundred thirty-four"

See h MyApp.Cldr.Number and h MyApp.Cldr.Number.to_string in iex for further information.

Localizing Lists

The Cldr.List module provides list formatting and is implemented in the ex_cldr_lists package. The public API for list formating is Cldr.List.to_string/2. Some examples:

iex> MyApp.Cldr.List.to_string(["a", "b", "c"], locale: "en")
"a, b, and c"

iex> MyApp.Cldr.List.to_string(["a", "b", "c"], locale: "en", format: :unit_narrow)
"a b c"

iex> MyApp.Cldr.List.to_string(["a", "b", "c"], locale: "fr")
"a, b et c"

See h MyApp.Cldr.List and h MyApp.Cldr.List.to_string in iex for further information.

Localizing Units

The Cldr.Unit module provides unit localization and is implemented in the ex_cldr_units package. The public API for unit localization is Cldr.Unit.to_string/3. Some examples:

iex> MyApp.Cldr.Unit.to_string 123, :gallon
"123 gallons"

iex> MyApp.Cldr.Unit.to_string 1234, :gallon, format: :long
"1 thousand gallons"

iex> MyApp.Cldr.Unit.to_string 1234, :gallon, format: :short
"1K gallons"

iex> MyApp.Cldr.Unit.to_string 1234, :megahertz
"1,234 megahertz"

iex> MyApp.Cldr.Unit.available_units
[:acre, :acre_foot, :ampere, :arc_minute, :arc_second, :astronomical_unit, :bit,
 :bushel, :byte, :calorie, :carat, :celsius, :centiliter, :centimeter, :century,
 :cubic_centimeter, :cubic_foot, :cubic_inch, :cubic_kilometer, :cubic_meter,
 :cubic_mile, :cubic_yard, :cup, :cup_metric, :day, :deciliter, :decimeter,
 :degree, :fahrenheit, :fathom, :fluid_ounce, :foodcalorie, :foot, :furlong,
 :g_force, :gallon, :gallon_imperial, :generic, :gigabit, :gigabyte, :gigahertz,
 :gigawatt, :gram, :hectare, :hectoliter, :hectopascal, :hertz, :horsepower,
 :hour, :inch, ...]

See h MyApp.Cldr.Unit and h MyApp.Cldr.Unit.to_string in iex for further information.

Localizing Dates

Formatting of relative dates and date times is supported in the Cldr.DateTime.Relative module implemented in the ex_cldr_dates_times package. The public API is MyApp.Cldr.DateTime.to_string/2 and MyApp.Cldr.DateTime.Relative.to_string/2. Some examples:

iex> MyApp.Cldr.Date.to_string Date.utc_today()
{:ok, "Aug 18, 2017"}

iex> MyApp.Cldr.Time.to_string Time.utc_now
{:ok, "11:38:55 AM"}

iex> MyApp.Cldr.DateTime.to_string DateTime.utc_now
{:ok, "Aug 18, 2017, 11:39:08 AM"}

iex> MyApp.Cldr.DateTime.Relative.to_string 1, unit: :day, format: :narrow
{:ok, "tomorrow"}

iex> MyApp.Cldr.DateTime.Relative.to_string(1, unit: :day, locale: "fr")
"demain"

iex> MyApp.Cldr.DateTime.Relative.to_string(1, unit: :day, format: :narrow)
"tomorrow"

iex> MyApp.Cldr.DateTime.Relative.to_string(1234, unit: :year)
"in 1,234 years"

iex> MyApp.Cldr.DateTime.Relative.to_string(1234, unit: :year, locale: "fr")
"dans 1 234 ans"

Gettext Pluralization

There is an experimental plurals module for Gettext called MyApp.Cldr.Gettext.Plural (where MyApp.Cldr is the name of your backend module). It is configured in Gettext by:

    defmodule MyApp.Gettext do
      use Gettext, otp_app: :my_app, plural_forms: MyApp.Cldr.Gettext.Plural
    end

MyApp.Cldr.Gettext.Plural will fall back to Gettext pluralisation if the locale is not known to Cldr. This module is only compiled if Gettext is configured as a dependency in your project.

Note that MyApp.Cldr.Gettext.Plural does not guarantee to return the same plural index as Gettext‘s own pluralization engine which can introduce some compatibility issues if you plan to mix plural engines.

Plugs

Cldr provides two plugs to aid integration into an HTTP workflow. These two plugs are:

See Cldr.Plug.SetLocale for a description of how to configure the plug.

In addition, note that when migrating from ex_cldr 1.x versions, a backend needs to be configured for both plugs. In the simplest case an example would be:

plug Cldr.Plug.SetLocale,
  apps:    [:cldr],
  cldr:    MyApp.Cldr

plug Cldr.Plug.AcceptLanguage,
  cldr_backend: MyApp.Cldr

Using Cldr.Plug.SetLocale without Phoenix

If you are using Cldr.Plug.SetLocale without Phoenix and you plan to use :path_param to identify the locale of a request then Cldr.Plug.SetLocale must be configured afterplug :match and beforeplug :dispatch. For example:

defmodule MyRouter do
  use Plug.Router

  plug :match

  plug Cldr.Plug.SetLocale,
    apps: [:cldr, :gettext],
    from: [:path, :query],
    gettext: MyApp.Gettext,
    cldr: MyApp.Cldr

  plug :dispatch

  get "/hello/:locale" do
    send_resp(conn, 200, "world")
  end
end

Using Cldr.Plug.SetLocale with Phoenix

If you are using Cldr.Plug.SetLocale with Phoenix and you plan to use the :path_param to identify the locale of a request then Cldr.Plug.SetLocale must be configured in the router module, not in the endpoint module. This is because conn.path_params has not yet been populated in the endpoint. For example:

defmodule MyAppWeb.Router do
  use MyAppWeb, :router

  pipeline :browser do
    plug :accepts, ["html"]
    plug :fetch_session
    plug Cldr.Plug.SetLocale,
        apps: [:cldr, :gettext],
        from: [:path, :query],
        gettext: MyApp.Gettext,
        cldr: MyApp.Cldr
    plug :fetch_flash
    plug :protect_from_forgery
    plug :put_secure_browser_headers
  end

  scope "/:locale", HelloWeb do
    pipe_through :browser

    get "/", PageController, :index
  end

end

About Language Tags

Note that ex_cldr defines locale strings according to the IETF standard as defined in RFC5646. ex_cldr also implements the u extension as defined in RFC6067 and the t extension defined in RFC6497. This is also the standard used by W3C.

The IETF standard is slightly different to the ISO/IEC 15897 standard used by Posix-based systems; primarily in that ISO 15897 uses a “_” separator whereas IETF and W3C use “-“.

Locale string are case insensitive but there are common conventions:

Sigil_l

As of ex_cldr version 2.23.0, a sigil is available to simplify creating t:Cldr.LanguageTag structs. Usage is:

iex> import Cldr.LanguageTag.Sigil
Cldr.LanguageTag.Sigil

# Returns a locale that is valid and known to
# the default backend module
iex> ~l(en-US)
#Cldr.LanguageTag<en-US [validated]>

# Same, but specifying the backend module
# MyApp.Cldr specifically
iex> ~l(en-US|MyApp.Cldr)
#Cldr.LanguageTag<en-US [validated]>

# The `u` flag will parse and validate
# the language tag but it may not be known
# as a configured locale
iex> ~l(zh)u
#Cldr.LanguageTag<zh [canonical]>

# Language tags can convey a lot more information
# than might be initially expected!
iex> ~l(en-u-ca-ethiopic-cu-aud-sd-gbsct-t-d0-lower-k0-extended-m0-ungegn-x-ux)
#Cldr.LanguageTag<en-t-d0-lower-k0-extended-m0-ungegn-u-ca-ethiopic-cu-aud-sd-gbsct-x-ux [validated]>

Locale extensions

Unicode defines the U extension which support defining the requested treatment of CLDR data formats. For example, a locale name can configure the requested:

For example, the following locale name will request the use of the timezone Australia/Sydney, and request the use of accounting format when formatting currencies:

iex> MyApp.Cldr.validate_locale "en-AU-u-tz-ausyd-cf-account"
{:ok,
 %Cldr.LanguageTag{
   canonical_locale_name: "en-Latn-AU",
   cldr_locale_name: "en-AU",
   extensions: %{},
   gettext_locale_name: "en",
   language: "en",
   language_subtags: [],
   language_variants: nil,
   locale: %Cldr.LanguageTag.U{cf: :account, timezone: "Australia/Sydney"},
   private_use: [],
   rbnf_locale_name: "en",
   requested_locale_name: "en-AU",
   script: :Latn,
   territory: :AU,
   transform: %{}
 }}

The implementation of these extensions is governed by each library in the ex_cldr family. As of January 2020, ex_cldr_numbers version 2.10 implements the following U extension keys:

Other libraries in the family will progressively implement other extension keys.

Notes

Developing ex_cldr

See the file DEVELOPMENT.md in the github repository.

Testing

Tests cover the full 566 locales defined in CLDR. Since Cldr attempts to maximize the work done at compile time in order to minimize runtime execution, the compilation phase for tests is several minutes.

Tests are run on Elixir 1.10 and later. ex_cldr may not run on Elixir versions before 1.10.