I'm trying to create a topic in a Kafka container running in Docker using a shell script file.

docker-compose.yml

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
    ports:
      - "22181:2181"

  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    depends_on:
      - zookeeper

    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

init.sh

#!/bin/zsh
docker compose up -d
docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test

When I run docker-compose the containers start-up fine, but the topic creation fails.

[2021-11-04 17:35:07,957] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

However, when I run the same commands in the command line, without any changes, it works fine and creates the topic.

Created topic Test.

I'm at a loss and reading previous questions on similar issues here hasn't help: mostly they were due to typos or adding quotation marks where they are not needed. I have tried changing the #! to bash, sh, and usr/bin/env (amongst others) to no avail.

Any help appreciated.

Docker version: version 20.10.8 Running on a Mac (Intel)

CodePudding user response：

Where/when are you running the init.sh script? Maybe the broker hasn't started yet.

Also, use the internal listener when you want to run commands inside the Docker network

docker-compose exec kafka \
  bash -c "kafka-topics --create --bootstrap-server kafka:9092 --partitions 1 --replication-factor 1 --topic Test"

CodePudding user response：

Modify init.sh

Run the container, docker-compose up -d, wait for the broker to run. check with the logs if the application is up and running.

Now init.sh should have only create the topic command

#!/bin/zsh
docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test1

docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test2

docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test3