Multi-broker Kafka Cluster Setup on your Local Development Machine with docker & docker-compose

TL;DR: a ready to use docker compose file: https://github.com/kaygorodov/multi-broker-local-kafka.

Usually when I test my applications locally, I use a single broker kafka cluster. Basically, I just run one docker container with kafka. However, recently I’ve had to test how an application behaves when one of the brokers is inaccessible. For that I had to setup a cluster with more than one broker. In this post, I will explain how the setup is done.

Problem

I want to have an easy way to bring up a multi-broker kafka cluster setup on my local machine.
Each broker should be accessible from the host machine.
Each broker should be accessible from other docker containers. _ Ideally, they should be accessible the same way from the host and containers.

Solution

Let’s recap first that any kafka broker has two settings which are important for us:

KAFKA_LISTENERS is a list of host/port pairs that the broker binds to.

KAFKA_ADVERTISED_LISTENERS is the information that is provided to clients (consumers or producers) pointing out what protocol/host/port they should use to connect to a given partition. For example, imagine that our client application queries a broker to get the metadata information for a given topic. As a result, the broker returns the metadata for each topic partition i.e. which protocol/host/port the client should use to connect to it. According to our prerequisites, this protocol/host/port should be accessible from within other containers as well as from the host machine.

We are going to expose all our containers with brokers to the host machine because we want to access them from the host machine. However, we cannot bind more than one container to the same port of the host machine, so we need to use different ports for different containers. For example, the first container with broker 1 will be bound to port 9092, the second container with broker 2 will be bound to port 9093, etc.

When we are accessing a broker from other containers we will use the container’s alias, e.g. kafka1, kafka2 etc. On the host machine, we will modify the /etc/hosts file and point all aliases to the localhost. This step is optional, and you can access your brokers via localhost. We will bind each kafka broker to a different port (by means of KAFKA_LISTENERS), so that we will be able to use the same port when we access docker from the host machine as well as when we want to access the broker from another container.

We are going to use the docker hub image built from this container: https://github.com/wurstmeister/kafka-docker. This project looks to be maintained, and lets us specify all required configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


version: "2"
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka1:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  kafka2:
    image: wurstmeister/kafka
    ports:
      - "9093:9093"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  kafka3:
    image: wurstmeister/kafka
    ports:
      - "9094:9094"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9094
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9094
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

On our host machine, we will add the following into the /etc/hosts file:

127.0.0.1 localhost kafka1 kafka2 kafka3

So, after setting up the cluster:

$ docker-compose up -V -d

Let’s create a topic (run the following from your host machine):

$ bin/kafka-topics.sh --create --bootstrap-server kafka1:9092 --replication-factor 3 --partitions 3 --topic test

And describe it (we can connector to any broker because all of them are accessible from the host machine):

$ bin/kafka-topics.sh --describe --bootstrap-server kafka2:9093 --topic test

Topic: test	PartitionCount: 3	ReplicationFactor: 3	Configs: segment.bytes=1073741824
	Topic: test	Partition: 0	Leader: 1003	Replicas: 1003,1002,1001	Isr: 1003,1002,1001
	Topic: test	Partition: 1	Leader: 1002	Replicas: 1002,1001,1003	Isr: 1002,1001,1003
	Topic: test	Partition: 2	Leader: 1001	Replicas: 1001,1003,1002	Isr: 1001,1003,1002

You can find a ready to use docker-compose file here: https://github.com/kaygorodov/multi-broker-local-kafka.