Cannot connect docker app to kafka docker container-CodePudding

I am trying to connect my Kafka container and my docker image all running locally and don't want to create a container with Kafka and zookeeper images.

When I am running the app without dockerizing it, it is working fine.

After running the dockerize app the docker logs is showing

could not read message dial tcp 172.21.0.3:9092: i/o timeout

in Kafka logs, it is showing

kafka-zookeeper-1  | 2021-12-07 06:17:15,755 [myid:1] - WARN  [NIOWorkerThread-7:ZooKeeperServer@1411] - Connection request from old client /172.18.0.1:56350; will be dropped if server is in r-o mode

this is the docker compose for kafka

version: "3"
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper
    volumes:
      - /Users/myuser/docker/volumes/kafka:/var/lib/kafka/data

Can anyone help, what am I doing wrong?

CodePudding user response：

When I am running the app without dockerizing it, it is working fine.

Basically, you want your app to connect to Kafka in both cases right?

when you're running your app locally (on the host machine)
and when you're running it as a dockerized application.

But you're only advertising one listener for 127.0.0.1:9092 (the host machine) so even if a dockerized client (your app) can access Kafka container, it will still fail to establish a connection because of a misleading listener configuration.

For example, I can use this for advertising two different listeners for two different networks (docker network and localhost on the host machine):

KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092

And when I run my dockerized application, it can connect to Kafka via broker:29092 and, similarly, when I run the app on the host machine, it can connect to Kafka via localhost:9092.

This post gives a more detailed explanation of how Kafka advertised listeners work and how should we configure them. It basically says:

You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. Otherwise they’ll try to connect to the internal host address–and if that’s not reachable then problems ensue.

CodePudding user response：

Here is your answer,

You must use host ip address. Do not use 127.0.0.1 or container ip address
Because, the brokers are advertising its ADVERTISED_LISTENERS to their clients, and clients try to connect to this addresses.
So, the ip addresses in the ADVERTISED_LISTENERS must accessible from outside container or other containers.
Refer following example

...
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://<hostip>:9092
...

CodePudding user response：

I am sharing one blog it may help you for more conceptual clarity if needed and the way environment variable KAFKA_CFG_ADVERTISED_LISTENERS is assigned.

blog link: Understanding Docker & Deploying Apache Kafka on docker in a few simple steps

KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092