whether apache Kafka 3.X cluster requires already started zookeeper or not-CodePudding

I am using a kafka cluster of three kafka nodes, and it is running on zookeeper cluster of nodes which is already up and running. Docker-compose is used with kafka docker image version 2.2.0 and zookeeper docker image version 3.5.

Now my plan is to use the upgraded version of kafka 3.x, where it is claimed that zookeeper is not required (may be explicitly not required). I was under impression that zookeeper will be started automatically. But I saw that Zookeeper is embedded in kafka and so I needed to start it also explicitly.

For that, I have to repackage kafka 3.x into two separate docker images. First image ZOOKEEPER_IMAGE, will call zookeeper’s zoo-start.sh through Docker file’s CMD command and it will set the zoo specific parameters.

Second image KAFKA_IMAGE will on contrary, will call kafka’s start.sh through Docker file’s CMD command and it will set the kafka specific parameters. CMD ["/start.sh"]

Also I can see that many kafka parameters are changed and at present I am getting this exception.

KafkaException: Unable to parse PLAINTEXT://127.0.0.1: to a broker endpoint -

Please suggest me if I am follow the correct approach and what is the solution of this exception.

Below is the code with one zookeeper and one Kafka node. Similar way in the original docker compose, it contains 3 kafka, 3 zookeeper nodes

zookeeper:
  hostname: ${HOST_IP}
  image: ${ZOOKEEPER_IMAGE}
  container_name: zookeeper
  command: /bin/bash -c "/start.sh"
  volumes:
    - ${VOLUMES_FOLDER}/zk/data:/data
    - ${VOLUMES_FOLDER}/zk/conf:/conf
  network_mode: "host"
  ports:
    - 0.0.0.0:${ZOOKEEPER_EXPOSED_PORT}:${ZOOKEEPER_EXPOSED_PORT}

kafka1:
  hostname: ${HOST_IP}
  image: ${KAFKA_IMAGE}
  depends_on:
  - zookeeper
  container_name: kafka1
  command: /bin/bash -c "/start.sh"
  volumes:
    - ${VOLUMES_FOLDER}/kf/1/logs:/logs
    - ${VOLUMES_FOLDER}/kf/1/data:/data
  environment:
    - KAFKA_ADVERTISED_HOST_NAME=${HOST_IP}
    - KAFKA_DELETE_TOPIC_ENABLE=true
    - KAFKA_BROKER_ID=10
    - port = ${KAFKA1_EXPOSED_PORT}
    - ZOOKEEPER_IP=${HOST_IP}
    - JMX_PORT=7208
  network_mode: "host"
  ports:
    - 0.0.0.0:${KAFKA1_EXPOSED_PORT}:${KAFKA1_EXPOSED_PORT}

CodePudding user response：

under impression that zookeeper will be started automatically

It won't. Kafka KRaft uses Raft protocol. Zookeeper is not required, at all, therefore is not started.

Zookeeper is embedded in kafka

It is not. Zookeeper server scripts are available, but they are not used in any embedded way, or automatically.

repackage kafka 3.x into two separate docker images

bitnami/kafka images, for example, already offer KRaft mode. Unclear what images you're currently using.

Your error is likely because advertised.host.name and port were both removed as Kafka properties. You should be using listeners and advertised.listeners.