ChDriver

A native-protocol DBConnection driver for ClickHouse -- it speaks ClickHouse's binary TCP protocol directly, rather than going over HTTP. It's usable on its own, independent of Ecto: ChDriver.start_link/1, ChDriver.query/2..4, ChDriver.query!/2..4, and ChDriver.stream/2..4 are the whole public surface. Everything else (ChDriver.DBConnection, ChDriver.Connection, ChDriver.Protocol) is wiring underneath it.

adapter (Ecto.Adapters.ClickHouse) builds an Ecto adapter on top of this driver -- Ecto.Adapters.ClickHouse.Connection implements Ecto.Adapters.SQL.Connection by driving a %ChDriver.Query{} through the normal DBConnection parse/encode/execute flow.

Usage

{:ok, pool} = ChDriver.start_link(hostname: "localhost", port: 9000, database: "default")

{:ok, %ChDriver.Result{columns: columns, rows: rows}} =
  ChDriver.query(pool, "SELECT number FROM system.numbers LIMIT 5")

# query!/2..4 raises instead of returning {:error, reason}
result = ChDriver.query!(pool, "SELECT number FROM system.numbers WHERE number > {min:UInt64} LIMIT 5",
  [{"min", "10", 1}])

# stream/2..4 yields one wire-protocol Data block at a time instead of
# buffering the whole result; `conn` must already be checked out via
# DBConnection.run/3 or DBConnection.transaction/3
{:ok, rows} =
  DBConnection.run(pool, fn conn ->
    conn
    |> ChDriver.stream("SELECT number FROM system.numbers LIMIT 200000")
    |> Enum.take(5)
  end)

start_link/1 accepts ChDriver.Connection.connect/1's connection options (:hostname, :port, :database, :username, :password, :connect_timeout, :recv_timeout, :max_buffer_size, :compression) plus the usual DBConnection.start_link/2 pool options (:pool_size, :name, etc.) -- it's a DBConnection pool of native-protocol connections, so anything that accepts a DBConnection reference works as conn.

query/2..4's params are {name, raw_text} or {name, raw_text, escape_rounds} tuples binding ClickHouse native {name:Type} placeholders written directly in statement -- see ChDriver.Protocol's and ChDriver.Query's moduledocs for the full parameter-binding story (Ecto.Adapters.ClickHouse.Connection's ?-based binding is built on top of this).

Compression: the compressed block envelope and its LZ4/CityHash NIF

ChDriver.Protocol.Block.Compressed (the pure-Elixir compressed-block wire envelope) and ChDriver.Codec (the Rust NIF backing it) together implement ClickHouse's compressed native-block wire envelope, and this driver wires it up via the :compression option (:none, the default, or :lz4) accepted by start_link/1 and overridable per call via query/4's/stream/4's opts. Enabling it negotiates LZ4 compression for both directions of a query's block traffic -- see ChDriver.Connection.connect/1 and ChDriver.Protocol.Messages.encode_query/2 for exactly how that negotiation works and what it means for callers.

Why a custom LZ4/CityHash codec

ClickHouse wraps every block it sends or receives in a fixed envelope:

[16 bytes] CityHash128 checksum (covers everything below)
[1 byte]   compression method marker (0x02 = NONE, 0x82 = LZ4, 0x90 = ZSTD)
[4 bytes]  compressed size, little-endian
[4 bytes]  uncompressed size, little-endian
[...]      payload

ChDriver.Protocol.Block.Compressed.encode/2 builds one of these envelopes (:lz4 or :none); decode/1 parses one off the front of a binary, returning the decompressed payload plus any unconsumed trailing bytes so callers can loop over back-to-back blocks. Multiple blocks are simply concatenated back-to-back with no additional framing.

Two things about this envelope make it awkward to build from off-the-shelf packages, which is why ChDriver.Codec exists as a small Rust NIF rather than reaching for a generic Hex dependency:

First, the LZ4 here isn't the LZ4 Frame format you get from the lz4 command line tool. It's the raw LZ4 block format, with no header of its own. ClickHouse supplies its own header instead (the four fields above).

Second, the checksum isn't today's CityHash. ClickHouse froze on CityHash v1.0.2 years ago -- a different, older algorithm than the v1.0.3 that most modern "cityhash" packages implement, not just a different byte-packing of the same hash. The two only agree for short inputs and diverge once the input exceeds roughly 64 bytes, so a generic "cityhash" library on your package manager of choice will silently give you the wrong answer for any real payload. You need the exact old version.

ChDriver.Codec handles both, and packs the checksum into the exact byte order ClickHouse expects on the wire:

compressed = ChDriver.Codec.lz4_compress(data)

# uncompressed_size has to come from somewhere -- ClickHouse's block
# header carries it, since the raw LZ4 format has no length prefix
{:ok, data} = ChDriver.Codec.lz4_decompress(compressed, uncompressed_size)

checksum = ChDriver.Codec.cityhash128(data)
# <<...16 bytes...>>

cityhash128/1 is checked against Google's own published CityHash v1.0.2 test vectors, so you're not just trusting our word for it.

Precompiled builds

ChDriver.Codec is a Rust NIF, and most people don't want to install a Rust toolchain just to use a ClickHouse client. Releases are built for the common targets (macOS and Linux, both gnu and musl, arm64 and x86_64) via rustler_precompiled, so a normal mix deps.get should just download a prebuilt binary rather than compiling anything.

If you're building from source, or on a target we don't precompile for, it'll fall back to compiling the crate locally. That needs cargo and a stable Rust toolchain on your machine.

Installation

Not on Hex. Pull it in from this repo with a path dependency:

def deps do
  [
    {:ch_driver, path: "path/to/clickhouse_adapter_elixir/ch_driver"}
  ]
end

Developing on the Rust NIF

mix deps.get
mix test
cd native/ch_driver_native && cargo test