Docker: How to connect to a kafka container which is not defined in docker-compose.yml file-CodePudding

I have 3 docker-compose files. One to start the kafka and the other two are consumer and producer. Added external_links in the other docker-compose files to kafka, but still unable to access kafka from inside containers. From outside the container, I can access through localhost:9092, but what about inside docker container.

# docker-compose1.yml
version: "3.6"
services:
  zookeeper:
    image: 'docker.io/bitnami/zookeeper:3.7'
    container_name: zookeeper
    ports:
      - '2181:2181'
    volumes:
      - 'zookeeper_data:/bitnami'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes

  kafka:
    image: 'docker.io/bitnami/kafka:3'
    container_name: kafka
    ports:
      - '9092:9092'
    volumes:
      - 'kafka_data:/bitnami'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_LISTENERS=PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_ADVERTISED_HOST_NAME=localhost
      - KAFKA_ADVERTISED_PORT=9092
      - KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
    depends_on:
      - zookeeper


volumes:
  zookeeper_data:
    external: true
  kafka_data:
    external: true

# docker-compose2.yml
version: "3.6"
services:
  web:
    hostname: ocp-transmitter
    image: 'ocp/transmitter'
    command: bash -c "bundle install && foreman start"
    ports:
      - '3000:3000'
    volumes:
      - .:/app:cached
    stdin_open: true
    tty: true
    external_links:
      - kafka

CodePudding user response：

First, remove these, they are deprecated

  - KAFKA_ADVERTISED_HOST_NAME=localhost
  - KAFKA_ADVERTISED_PORT=9092

Second, read the bitnami image documentation more carefully, all the Kafka properties start with KAFKA_CFG_, then read the section about internal/external listeners

The linked answer(s) are correct Communication between multiple docker-compose projects

Run docker network create with a name to setup an external bridge network separately from Compose, then add networks section to each service in that network (Zookeeper, Kafka, and your Kafka clients). Then make sure it's external

networks:
  example-net:
    external: true

Then you'd use kafka:29092 in your apps, not localhost, and not port 9092