Brucke - Inter-cluster bridge of kafka topics

Brucke is a Inter-cluster bridge of kafka topics powered by Brod

Brucke bridges messages from upstream topic to downstream topic with configurable re-partitionning strategy.

Configuration

A brucke config file is a YAML file.

Config file path should be set in config_file variable of brucke app config, or via BRUCKE_CONFIG_FILE OS env variable.

Cluster names and client names must comply to erlang atom syntax.

kafka_clusters:
  kafka_cluster_1:
    - "localhost:9092"
  kafka_cluster_2:
    - "kafka-1:9092"
    - "kafka-2:9092"
brod_clients:
  - client: brod_client_1
    cluster: kafka_cluster_1
    config: [] # optional
routes:
  - upstream_client: brod_client_1
    downstream_client: brod_client_1
    upstream_topics:
      - "topic_1"
    downstream_topic: "topic_2"
    repartitioning_strategy: strict_p2p
    default_begin_offset: earliest # optional
    filter_module: brucke_filter # optional

Options for repartitioning strategy

NOTE: For compacted topics, strict_p2p is the only choice.

• key_hash: hash the message key to downstream partition number
• strict_p2p: strictly map the upstream partition number to downstream partition number, worker will refuse to start if upstream and downstream topic has different number of partitions
• random: randomly distribute upstream messages to downstream partitions

Customized Message Filtering and or Transformation

Implement brucke_filter behaviour to have messages filtered and or transformed before produced to downstream topic.

Graphite reporting

If the following app config variables are set, brucke will send metrics to a configured graphite endpoint:

• graphite_root_path: a prefix for metrics, e.g "myservice"
• graphite_host: e.g. "localhost"
• graphite_port: e.g. 2003

Alternatively, you can use corresponding OS env variables:

• BRUCKE_GRAPHITE_ROOT_PATH
• BRUCKE_GRAPHITE_HOST
• BRUCKE_GRAPHITE_PORT

RPM packaging

Generate a release and package it into an rpm package:

make rpm

Brucke package installs release and creates corresponding systemd service. Config files are in /etc/brucke, OS env can be set at /etc/sysconfig/brucke, logs are in /var/log/brucke.

Operating via systemctl:

systemctl start brucke
systemctl enable brucke
systemctl status brucke

Http endpoint

Default port is 8080, customize via http_port config option or via BRUCKE_HTTP_PORT OS env variable.

GET /ping

Returns pong if the application is up and running.

GET /healthcheck

Responds with status 200 if everything is OK, and 500 if something is not OK.
Also returns healthy and unhealthy routes in response body in JSON format.

Example response:

{
    "discarded": [
        {
            "downstream": {
                "endpoints": [
                    {
                        "host": "localhost",
                        "port": 9192
                    }
                ],
                "topic": "brucke-filter-test-downstream"
            },
            "options": {
                "default_begin_offset": "earliest",
                "filter_init_arg": [],
                "filter_module": "brucke_test_filter",
                "repartitioning_strategy": "strict_p2p"
            },
            "reason": [
                "filter module brucke_test_filter is not found\nreason:embedded\n"
            ],
            "upstream": {
                "endpoints": [
                    {
                        "host": "localhost",
                        "port": 9092
                    }
                ],
                "topics": [
                    "brucke-filter-test-upstream"
                ]
            }
        }
    ],
    "healthy": [
        {
            "downstream": {
                "endpoints": [
                    {
                        "host": "localhost",
                        "port": 9192
                    }
                ],
                "topic": "brucke-basic-test-downstream"
            },
            "options": {
                "consumer_config": {
                    "begin_offset": "earliest"
                },
                "filter_init_arg": [],
                "filter_module": "brucke_filter",
                "max_partitions_per_group_member": 12,
                "producer_config": {
                    "compression": "no_compression"
                },
                "repartitioning_strategy": "strict_p2p"
            },
            "upstream": {
                "endpoints": [
                    {
                        "host": "localhost",
                        "port": 9092
                    }
                ],
                "topics": "brucke-basic-test-upstream"
            }
        },
        {
            "downstream": {
                "endpoints": [
                    {
                        "host": "localhost",
                        "port": 9092
                    }
                ],
                "topic": "brucke-test-topic-2"
            },
            "options": {
                "consumer_config": {
                    "begin_offset": "earliest"
                },
                "filter_init_arg": [],
                "filter_module": "brucke_filter",
                "max_partitions_per_group_member": 12,
                "producer_config": {
                    "compression": "no_compression"
                },
                "repartitioning_strategy": "strict_p2p"
            },
            "upstream": {
                "endpoints": [
                    {
                        "host": "localhost",
                        "port": 9092
                    }
                ],
                "topics": "brucke-test-topic-1"
            }
        },
        {
            "downstream": {
                "endpoints": [
                    {
                        "host": "localhost",
                        "port": 9092
                    }
                ],
                "topic": "brucke-test-topic-3"
            },
            "options": {
                "consumer_config": {
                    "begin_offset": "latest"
                },
                "filter_init_arg": [],
                "filter_module": "brucke_filter",
                "max_partitions_per_group_member": 12,
                "producer_config": {
                    "compression": "no_compression"
                },
                "repartitioning_strategy": "key_hash"
            },
            "upstream": {
                "endpoints": [
                    {
                        "host": "localhost",
                        "port": 9092
                    }
                ],
                "topics": "brucke-test-topic-2"
            }
        },
        {
            "downstream": {
                "endpoints": [
                    {
                        "host": "localhost",
                        "port": 9192
                    }
                ],
                "topic": "brucke-test-topic-1"
            },
            "options": {
                "consumer_config": {
                    "begin_offset": "latest"
                },
                "filter_init_arg": [],
                "filter_module": "brucke_filter",
                "max_partitions_per_group_member": 12,
                "producer_config": {
                    "compression": "no_compression"
                },
                "repartitioning_strategy": "random"
            },
            "upstream": {
                "endpoints": [
                    {
                        "host": "localhost",
                        "port": 9192
                    }
                ],
                "topics": "brucke-test-topic-3"
            }
        }
    ],
    "status": "failing",
    "unhealthy": []
}