README.md

# Db2Kafka

[![Build Status](https://travis-ci.org/PagerDuty/db2kafka.svg?branch=master)](https://travis-ci.org/PagerDuty/db2kafka)

This service takes records from a database's `outbound_kafka_queue` table and
ships them to Kafka.

## Development


### Setup

#### Elixir

Short version: `iex -S mix` to run the service in development; `mix test --only unit --no-start` to run unit tests

#### MySQL

Setup the various database environment variables as per config/ before proceeding. You need a table called `outbound_kafka_queue` in your database:

```sql
CREATE TABLE `outbound_kafka_queue` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `topic` varchar(128) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `item_key` varchar(128) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `item_body` varbinary(60000) DEFAULT NULL,
  `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

#### Kafka

Install [Kafka](http://kafka.apache.org/) if you don't already have it running!

Setup the various Kafka environment variables as per config/ before proceeding.

### Running Tests

#### Static Analysis

        mix dialyzer

Resolve all the warnings before submitting a PR!

To speed up `dialyzer`, build a Persistent Lookup Table (PLT):

        mix dialyzer.plt

#### Unit Tests

        mix test --only unit --no-start

#### Integration Tests

- Create MySQL database (see above for schema);

- Create Kafka topic:

        kafka-topics.sh --zookeeper 127.0.0.1:2181 --create --topic foo --partitions 1 --replication-factor 1


- Run Integration Tests

        mix test --only integration --no-start