Increasing number of brokers in kafka using KAFKA_ADVERTISED_HOST

I'm new to Kafka and I'm trying to run a Kafka service on my local machine and use it to transfer some data from one .NET project to another. I'm using docker-compose.yml file to create two docker containers for zookeeper and Kafka from wurstmeister. In Kafka definition in environment variables there is KAFKA_ADVERTISED_HOST_NAME which I set to 127.0.0.1. In docker hub of wurstmeister/kafka says I quote

"modify the KAFKA_ADVERTISED_HOST_NAME in docker-compose.yml to match your docker host IP (Note: Do not use localhost or 127.0.0.1 as the host IP if you want to run multiple brokers.)".

When I set my topic to have more than 1 replica and more than 1 partition I get this message

"Error while executing topic command : Replication factor: 2 larger than available brokers: 1.".

What is the right IP address to define in KAFKA_ADVERTISED_HOST_NAME which will allow me to get more than 1 broker?

version: '2'
services:
 zookeeper:
  image: wurstmeister/zookeeper
  ports:
   - "2181:2181"
 kafka:
  image: wurstmeister/kafka
  ports:
   - "9092:9092"
  environment:
   KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
   KAFKA_CREATE_TOPICS: "simpletalk_topic:2:2"
   KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  volumes:
   - /var/run/docker.sock:/var/run/docker.sock

CodePudding user response：

Note - Running multiple brokers on one machine does not provide true fault tolerance.

use it to transfer some data from one .NET project to another

You only need one broker for this.

First, suggest reading https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

KAFKA_ADVERTISED_HOST_NAME is deprecated. Don't use it.

It has been replaced by KAFKA_ADVERTISED_LISTENERS, which can both contain ${DOCKER_HOST_IP:-127.0.0.1} (or your host IP from ifconfig) since this is the IP your client will use (from the host).
Since brokers are clients of themselves, you also need to advertise the container names. And, if you want to containerize your client, those addresses would also be used by your app.

Beyond Kafka networking configs, KAFKA_BROKER_ID needs to be different between each, and your error is suggesting you need to override all the other replication factor broker configs.

All in all.

version: '3'
services:
 zookeeper:
  image: wurstmeister/zookeeper
 kafka-1:
  image: wurstmeister/kafka
  depends_on: [zookeeper]
  ports:
   - "9092:9092"
  environment:
   KAFKA_BROKER_ID: 1
   KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${DOCKER_HOST_IP:-127.0.0.1}:9092,PLAINTEXT_INTERNAL://kafka-1:29092
   KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_INTERNAL://:29092
   KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT_INTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
   KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT_INTERNAL
   KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
   KAFKA_DEFAULT_REPLICATION_FACTOR: 2
   KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
   KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
 kafka-2:
  image: wurstmeister/kafka
  ports:
   - "9093:9093"
  environment:
   KAFKA_BROKER_ID: 2  # unique
   KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${DOCKER_HOST_IP:-127.0.0.1}:9093,PLAINTEXT_INTERNAL://kafka-2:29093
   KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093,PLAINTEXT_INTERNAL://:29093
   KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT_INTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
   KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT_INTERNAL
   KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
   KAFKA_DEFAULT_REPLICATION_FACTOR: 2
   KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
   KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2
   KAFKA_CREATE_TOPICS: "simpletalk_topic:2:2"
  depends_on:  # ensure this joins the other
   - zookeeper
   - kafka-1

I'm trying to run a Kafka service on my local machine

Set bootstrap.servers="localhost:9092,localhost:9093"

Further reading - Connect to Kafka running in Docker

Example usage

Note that topic creation doesn't seem to automatically work. You should ideally be using your application to actually create / check topic existence.

$ kcat -b localhost:9093 -L
Metadata for all topics (from broker -1: localhost:9093/bootstrap):
 2 brokers:
  broker 2 at 127.0.0.1:9093
  broker 1 at 127.0.0.1:9092 (controller)
 0 topics:

producing some data... and consuming it

$ kcat -b localhost:9093 -t test -o earilest
% Auto-selecting Consumer mode (use -P or -C to override)
hello
world
sample
data
of
no
importance

CodePudding user response：

it should be ip address of your machine. Linux: use ifconfig command and your IP will be inet <IP_ADDRESS> in windows you will get this by using ipconfig

CodePudding user response：

Look at this, it has what you need : https://github.com/conduktor/kafka-stack-docker-compose