Kayrock

CI Checks CI Tests Module Version Hex Docs Total Download License Last Updated

Idiomatic Elixir interface to the Kafka protocol.

This library provides serialization and deserialization for all Kafka protocol messages, based on kafka_protocol for Erlang. It is built to work with KafkaEx though there is no reason why it couldn't be used elsewhere.

Installation

Add kayrock to your list of dependencies in mix.exs:

def deps do
  [
    {:kayrock, "~> 1.1"}
  ]
end

Requirements

Upgrading from 1.0.x: 1.1.0 reports the Kafka 4.4.0 maximum versions from KafkaSchemaMetadata.version_range/1. If you use Kayrock through KafkaEx, upgrade to KafkaEx 1.1.3 or later first: it caps negotiation at the versions it implements. KafkaEx <= 1.1.2 would start negotiating versions it cannot build.

Basic Architecture and Usage

Kayrock generates structs for every version of every message in the Kafka protocol. It does this by converting the Erlang-based schema descriptions in :kpro_schema from the kafka_protocol library and is therefore limited to messages and versions included there.

Each request message has a serializer (that generates iodata), and each response message has a deserializer. There is also a protocol, Kayrock.Request, that has an implementation created for each request.

The generated structs are namespaced as follows: Kayrock.<API>.V<version>.[Request|Response]

For example:

Example Usage

alias Kayrock.Request

# Create a request
request = %Kayrock.ApiVersions.V1.Request{
  client_id: "my_client",
  correlation_id: 0
}

# Serialize to wire protocol (iodata)
wire_protocol = Request.serialize(request)

# ... send wire_protocol over your connection ...
# ... receive binary response ...

# Deserialize the response
response_deserializer = Request.response_deserializer(request)
{deserialized_resp, _rest} = response_deserializer.(binary_response)

# deserialized_resp is now a %Kayrock.ApiVersions.V1.Response{}

Message Compression

Kayrock supports four compression formats:

Format Built-in Dependency Required
gzip Yes None
snappy No {:snappyer, "~> 1.2"}
lz4 No {:lz4b, "~> 0.0.13"}
zstd OTP 28+ {:ezstd, "~> 1.0"} (for OTP < 28)

Installing Compression Dependencies

Add the compression libraries you need to your mix.exs:

def deps do
  [
    {:kayrock, "~> 1.1"},

    # Add compression libraries as needed:
    {:snappyer, "~> 1.2"},     # For Snappy compression
    {:lz4b, "~> 0.0.13"},      # For LZ4 compression
    {:ezstd, "~> 1.0"},        # For Zstandard (OTP < 28)
  ]
end

Zstandard Support

Zstandard compression is available via:

  1. OTP 28+: Native :zstd module (no dependency needed)
  2. OTP < 28: Add {:ezstd, "~> 1.0"} to your dependencies

Code Generation

This repo includes a mix task, mix gen.kafka_protocol, that produces the code in lib/generated. The generated code is checked into the repository to simplify usage. End users should not need to run the mix task unless they are doing development on Kayrock itself.

Schemas come from two sources:

In JSON-derived versions, KIP-482 tagged fields are ordinary struct fields: a field left at its default is not sent, and a decoded response fills them in (with their defaults when absent). Tags Kayrock doesn't know about are kept as raw {tag, binary} pairs in tagged_fields, in both directions. Versions sourced from kpro_schema keep every tagged field in tagged_fields, as before.

Which APIs are generated, and up to which version, is decided by one table: Kayrock.Generate.ApiTable (lib/kayrock/generate/api_table.ex). Supporting a newer version of an API means raising its maximum there and running mix gen.kafka_protocol. The task validates the table first and refuses versions that are marked latestVersionUnstable in the JSON specs. mix gen.kafka_protocol --check (run in CI) fails if regenerating would change anything in lib/generated.

Relationship to Other Libraries

Kayrock uses only the kpro_schema module from kafka_protocol, and only at code-generation time, for the API versions it has always generated. Newer versions are generated from Apache Kafka's own JSON message definitions. kafka_protocol provides quite a lot of functionality (especially when used as part of brod). This repo limits the integration surface because:

  1. kafka_protocol produces Erlang records whereas Kayrock provides Elixir structs
  2. kafka_protocol provides network-level implementation whereas Kayrock only provides serialization and deserialization
  3. kpro_schema is itself generated from Kafka's JSON message definitions, which Kayrock now reads directly for newer versions

Built-in Client (Development Only)

This repo includes a lightweight Kafka client implementation for development and testing purposes.

Warning: The built-in client (Kayrock.Client) is NOT production-ready. For production usage, please use:

Testing

Kayrock includes both unit tests and integration tests.

Unit Tests

mix test

Integration Tests

Integration tests require Docker. We use testcontainers to run a Kafka cluster.

# Requires Docker to be running
mix test.integration

Kafka 4.x Tests

mix test.kafka4 runs the :kafka4 suite (test/kafka4/) against a single-node KRaft apache/kafka:4.4.0 broker (Kayrock.Kafka4Container). The :integration, :sanity and :chaos suites keep using Confluent 7.4. Kafka 4.x removed many old protocol versions (KIP-896), and the broker does not answer requests at those versions. test/kafka4/api_versions_smoke_test.exs lists which generated versions 4.4.0 no longer accepts.

# Requires Docker to be running
mix test.kafka4

Contributing

See CONTRIBUTING.md for development setup and guidelines.

License

MIT License - see LICENSE.txt for details.