Kafka container connecting from inside the container and outside using the same hostname-CodePudding

I am aware this topic has several questions and blog posts about it. I am following these two: https://rmoff.net/2018/08/02/kafka-listeners-explained/

https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/ But unfortunately, without success. I'm trying to make it so the same code will work whether I'm running it from my IDE where the kafka client is in a container, or whether the code I'm running is in a container within the network. I am able to make each scenario work on its own, but not the two together. My docker compose:

  zoo1:
    image: confluentinc/cp-zookeeper:7.2.1
    hostname: zoo1
    container_name: zoo1
    ports:
     - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_SERVERS: zoo1:2888:3888

  kafka1:
    image: confluentinc/cp-kafka:7.2.1
    hostname: kafka1
    container_name: kafka1
    ports:
      - "9092:9092"
      - "29092:29092"
    environment:
      KAFKA_LISTENERS: INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
    depends_on:
      - zoo1

In this docker-compose, communication within the docker network using kafka1:29092 as bootstrap works great. but, from my laptop using the same doesn't work. Is there anyway to ensure that both locally and inside the container network I can bootstrap to kafka1:29092? Do I even need the external listener? Thanks

CodePudding user response：

anyway to ensure that both locally and inside the container network I can bootstrap to kafka1:29092?

No.

Your host isn't aware of the DNS / service names used by Docker.

Instead, add an environment variable in your code like KAFKA_BOOTSTRAP_SERVERS, then set that as a variable in your IDE (as localhost:9092) via a run config, or as a container variable (kafka1:29092)

You can also remove - "29092:29092" from your compose file since your host will never need that port to connect with the broker