CC Precompiler

C/C++ Cross-compiler Precompiler is a library that supports elixir_make‘s precompilation feature. It’s customisble and easy to extend.

The guide for how to cc_precompiler can be found in the PRECOMPILATION_GUIED.md file.

Installation

If available in Hex, the package can be installed by adding cc_precompiler to your list of dependencies in mix.exs:

def deps do
  [
    {:cc_precompiler, "~> 0.1.1"}
  ]
end

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/cc_precompiler.

Default Targets

By default, it will probe some well-known C/C++ crosss-compilers existing on your system:

Linux

cc_precompiler will try to find #{prefix}gcc in $PATH, and if #{prefix}gcc can be found, then the correspondong target will be activiated. Otherwise, that target will be ignored.

macOS

cc_precompiler will try to find gcc in $PATH, and if gcc can be found, then both x86_64 and arm64 target will be activiated. Otherwise, both targets will be ignored.

Note

Triplet for current host will be always available, :erlang.system_info(:system_architecture).

For macOS targets, the version part will be trimmed, e.g., x86_64-apple-darwin21.6.0 will be x86_64-apple-darwin.

Note

Conditionally switch on/off compilation flags depending on the target

During the compilation, cc_precompiler will set and update the environment variable CC_PRECOMPILER_CURRENT_TARGET to the current target’s triplet.

The reason we might need this is that some 3rd party library may support some feature, like AVX, but they do not offer an auto-detection mechanism, and we have to manually switch on/off corresponding compilation flags.

An example with further explanation can be found on cocoa-xu/nif_opt_flags.

Last but not least, as the name suggests, this environment variable is set by cc_precompiler, thus if you switch to another precompiler, please check their manual for the equvilent.

Customise Precompilation Targets

To override the default configuration, please set the cc_precompiler key in project. For example,

def project do
[ 
  # ...
  cc_precompiler: [
    # optional config key
    #   false - target triplet for the current machine will be included in all available targets
    #   true  - only targets listed in `compilers` will be included in all available targets
    # defaults to `false`
    only_listed_targets: true,

    # optional config key
    # clean up the priv directory between different targets
    # 
    # for example, common assets for different targets can stay
    # in the `priv` directory (instead of copying/downloading them
    # multiple times)
    # but target specific assets or .o files should be cleaned
    # so that `make` can compile/generate these files for the next target
    #
    # the value for `cleanup` should be a string indicating the cleanup target
    # in the makefile.
    # 
    # for example, cc_precompiler will call `make mycleanup` between each build
    # if the value for the key `cleanup` is set to `mycleanup`
    #
    # also, cc_precompiler will stop if `make mycleanup` exited with non-zero code
    #
    # the default value for this key is `nil`, and in such case, cc_precompiler 
    # will not do anything between each build
    cleanup: "mycleanup",

    # optional config that provides a map of available compilers
    # on different systems
    compilers: %{
      # key (`:os.type()`)
      #   this allows us to provide different available targets 
      #   on different systems
      # value is a map that describes which compilers are available
      #
      # key == {:unix, :linux} => when compiling on Linux
      {:unix, :linux} => %{
        # key (target triplet) => `riscv64-linux-gnu`
        # value => `PREFIX`
        #   - for strings, the string will be used as the prefix of
        #         the C and C++ compiler respectively, i.e.,
        #         CC=`#{prefix}gcc`
        #         CXX=`#{prefix}g++`
        "riscv64-linux-gnu" => "riscv64-linux-gnu-",
        # key (target triplet) => `armv7l-linux-gnueabihf`
        # value => `{CC, CXX}`
        #   - for 2-tuples, the elements are the executable name of
        #         the C and C++ compiler respectively
        "armv7l-linux-gnueabihf" => {
          "arm-linux-gnueabihf-gcc",
          "arm-linux-gnueabihf-g++"
        },
        # key (target triplet) => `armv7l-linux-gnueabihf`
        # value => `{CC_EXECUTABLE, CXX_EXECUTABLE, CC_TEMPLATE, CXX_TEMPLATE}`
        #
        # - for 4-tuples, the first two elements are the same as in
        #       2-tuple, the third and fourth elements are the template
        #       string for CC and CPP/CXX. for example,
        #       
        #       the last entry below shows the example of using zig as the
        #       crosscompiler for `aarch64-linux-musl`, 
        #       the "CC" will be
        #           "zig cc -target aarch64-linux-musl", 
        #       and "CXX" and "CPP" will be
        #           "zig c++ -target aarch64-linux-musl"
        "aarch64-linux-musl" => {
          "zig", 
          "zig", 
          "<% cc %> cc -target aarch64-linux-musl", 
          "<% cxx %> c++ -target aarch64-linux-musl"
        }
      },
      # key == {:unix, :darwin} => when compiling on macOS
      {:unix, :darwin} => %{
        # key (target triplet) => `aarch64-apple-darwin`
        # value => `{CC, CXX}`
        "aarch64-apple-darwin" => {
          "gcc -arch arm64", "g++ -arch arm64"
        },
        # key (target triplet) => `aarch64-linux-musl`
        # value => `{CC_EXECUTABLE, CXX_EXECUTABLE, CC_TEMPLATE, CXX_TEMPLATE}`
        "aarch64-linux-musl" => {
          "zig",
          "zig",
          "<% cc %> cc -target aarch64-linux-musl",
          "<% cxx %> c++ -target aarch64-linux-musl"
        },
        # key (target triplet) => `my-custom-target`
        # - for 3-tuples, the first element should be `:script`
        #       the second element is the path to the elixir script file
        #       the third element is a 2-tuple, 
        #          the first one is the name of the module
        #          the second one is custom args
        #       the module need to impl the `compile/5` callback declared in 
        #          `CCPrecompiler.CompilationScript`
        "my-custom-target" => {
          :script, "custom.exs", {CustomCompile, []}
        },
        # key (target triplet) => `macos-universal`
        # on macOS, CCPrecompiler also provides a builtin module to create 
        # universal binary for NIF libraries that only has a `nif.so` file
        "macos-universal" => {
          :script, "", {CCPrecompiler.UniversalBinary, []}
        }
      }
    }
  ]
]

CCPrecompiler.CompilationScript is defined as follows,

defmodule CCPrecompiler.CompilationScript do
  @callback compile(
              app :: atom(),
              version :: String.t(),
              nif_version :: String.t(),
              target :: String.t(),
              command_line_args :: [String.t()],
              custom_args :: [String.t()]
            ) :: :ok | {:error, String.t()}
end

Custom Compilation Script Examples

Compile with ccache

defmodule CCPrecompiler.CCache do
  @moduledoc """
  Compile with ccache

  ## Example

    "x86_64-linux-gnu" => {
      :script, "custom.exs", {CCPrecompiler.CCache, []}
    }

  It's also possible to do this using a 4-tuple:

    "x86_64-linux-musl" => {
      "gcc", "g++", "ccache <% cc %>", "ccache <% cxx %>"
    }

  """

  @behaviour CCPrecompiler.CompilationScript

  @impl CCPrecompiler.CompilationScript
  def compile(app, version, nif_version, target, args, _custom_args) do
    System.put_env("CC", "ccache gcc")
    System.put_env("CXX", "ccache g++")
    System.put_env("CPP", "ccache g++")

    ElixirMake.Precompiler.mix_compile(args)
  end
end

Build A Universal NIF Binary on macOS

File can be found at lib/complation_script/universal_binary.ex.

defmodule CCPrecompiler.UniversalBinary do
  @moduledoc """
  Build a universal binary on macOS

  ## Example

    "macos-universal" => {
      :script, "universal_binary.exs", {CCPrecompiler.UniversalBinary, []}
    }

  """

  @behaviour CCPrecompiler.CompilationScript

  @impl CCPrecompiler.CompilationScript
  def compile(_app, _version, _nif_version, _target, args, _custom_args) do
    config = Mix.Project.config()
    app_priv = Path.join(Mix.Project.app_path(config), "priv")
    make_precompiler_filename = config[:make_precompiler_filename] || "nif"
    nif_file = "#{make_precompiler_filename}.so"

    compiled_bin = Path.join(app_priv, nif_file)
    x86_64_bin = Path.join(app_priv, "#{make_precompiler_filename}_x86_64.so")
    aarch64_bin = Path.join(app_priv, "#{make_precompiler_filename}_aarch64.so")

    File.rm(compiled_bin)

    # first we compile `x86_64-apple-darwin`
    :ok = System.put_env("CC", "gcc -arch x86_64")
    System.put_env("CXX", "gcc -arch x86_64")
    System.put_env("CPP", "g++ -arch x86_64")
    ElixirMake.Compiler.compile(args)
    File.rename!(compiled_bin, x86_64_bin)

    # then we compile `aarch64-apple-darwin`
    System.put_env("CC", "gcc -arch arm64")
    System.put_env("CXX", "gcc -arch arm64")
    System.put_env("CPP", "g++ -arch arm64")
    ElixirMake.Compiler.compile(args)
    File.rename!(compiled_bin, aarch64_bin)

    {%IO.Stream{}, exit_status} = System.cmd("lipo", ["-create", "-output", compiled_bin, x86_64_bin, aarch64_bin])

    File.rm!(x86_64_bin)
    File.rm!(aarch64_bin)

    if exit_status == 0 do
      :ok
    else
      Mix.raise("Failed to create universal binary")
    end
  end
end