How do I create a Kafka topic using a docker-compose file?-CodePudding

I'm new to Kafka and I am trying to create a demo Kafka server and work with it.

I have already created Zookeeper and Kafka containers using a docker-compose file and they are started and running fine.

If I docker exec into the Kafka container and run:

kafka-topics --create --zookeeper zookeeper-1 --replication-factor 1 --partitions 1 --topic demo-topic

the topic is created successfully... But I would like to spin up the Kafka broker and then programmatically create the topics without user interaction (Will ultimately be for a pipeline)

I've tried two different Kafka images (other was confluentinc/cp-kafka)... I have also tried changing bash to sh under kafka-1.command

Could really do with some help here. Below is my docker-compose file and the error response I'm getting in my terminal for bitnami image & for confluent image.

ERROR (BITNAMI)
/opt/bitnami/scripts/kafka/entrypoint.sh: line 27: exec: bash -c "kafka-topics --create --zookeeper zookeeper-1 --replication-factor 1 --partitions 1 --topic demo-topic": not found

ERROR (CONFLUENT)
Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "bash -c \"kafka-topics --create --zookeeper zookeeper-1 --replication-factor 1 --partitions 1 --topic demo-topic\"": executable file not found in $PATH: unknown

DOCKER-COMPOSE.YAML

services:
  zookeeper-1:
    container_name: zookeeper-1
    image: zookeeper
    restart: always
    ports:
      - 2181:2181
    environment:
      - ZOOKEEPER_CLIENT_PORT=2181
    volumes: 
    - ./config/zookeeper-1/zookeeper.properties:/kafka/config/zookeeper.properties
  kafka-1:
    container_name: kafka-1 
    image: bitnami/kafka
    depends_on: 
      - zookeeper-1
    ports: 
      - 29092:29092
      - 9092:9092
    environment:
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper-1:2181
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:29092
      - ALLOW_PLAINTEXT_LISTENER=yes
    command:
      - bash -c "kafka-topics --create --zookeeper zookeeper-1 --replication-factor 1 --partitions 1 --topic demo-topic"

CodePudding user response：

For both errors, the command you are providing is being appended to the container's entrypoints, which do not accept bash commands... You would need to override both the entrypoint and the command, as well as give the full path to the kafka-topics script since it is not on the $PATH.

However, you cannot do this with the command on the same container as the broker because that overrides the command that actually starts the server.

You will need a secondary "init container" that creates the topics , but it would be easier for you to write this topic creation logic into your own producer/consumer applications with an AdminClient.createTopics call (assuming Java).

Otherwise, you can use wurstmeister/kafka image which has an environment variable KAFKA_CREATE_TOPICS for this purpose, or just use docker-compose exec kafka-1 "..." after the containers are up