Db2Kafka

This service takes records from a database’s outbound_kafka_queue table and ships them to Kafka.

Development

Setup

Elixir

Short version: iex -S mix to run the service in development; mix test --only unit --no-start to run unit tests

MySQL

Setup the various database environment variables as per config/ before proceeding. You need a table called outbound_kafka_queue in your database:

CREATE TABLE `outbound_kafka_queue` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `topic` varchar(128) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `item_key` varchar(128) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `item_body` varbinary(60000) DEFAULT NULL,
  `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

Kafka

Install Kafka if you don’t already have it running!

Setup the various Kafka environment variables as per config/ before proceeding.

Running Tests

Static Analysis

    mix dialyzer

Resolve all the warnings before submitting a PR!

To speed up dialyzer, build a Persistent Lookup Table (PLT):

    mix dialyzer.plt

Unit Tests

    mix test --only unit --no-start

Integration Tests

• Create MySQL database (see above for schema);

• Create Kafka topic:

    kafka-topics.sh --zookeeper 127.0.0.1:2181 --create --topic foo --partitions 1 --replication-factor 1

• Run Integration Tests

    mix test --only integration --no-start