Unable to start debizium kafka broker with internal and external listeners-CodePudding

docker run -it --rm --name kafka -p 9092:9092 -p 9094:9094 --link zookeeper:zookeeper --env KAFKA_LISTENERS=LISTENER_INTERNAL://:9092,LISTENER_EXTERNAL://:9094 --env KAFKA_ADVERTISED_LISTENERS=LISTENER_INTERNAL://kafka:9092,LISTENER_EXTERNAL://localhost:9094 --env KAFKA_ALLOW_PLAINTEXT_LISTENER="yes" --env KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=LISTENER_INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT --env KAFKA_INTER_BROKER_LISTENER_NAME=LISTENER_INTERNAL  debezium/kafka:1.9

I am trying to start Kafka via debezium's docker image. I have KafkaConsumers both within the docker network and outside the docker network.

If I start Kafka with

--env KAFKA_ADVERTISED_LISTENERS=LISTENER_INTERNAL://:9092,LISTENER_EXTERNAL://localhost:9094

My external consumers work fine by connecting to port 9094.

Now my docker internal consumers don't work if LISTENER_INTERNAL is set to :9092. All examples I have seen use docker compose and they do LISTENER_INTERNAL://kafka:9092

If I try that (original snippet), then I get the following errors and they refuse to start.

java.net.UnknownHostException: kafka
    at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
    at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
    at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:511)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:468)
    at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173)
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:988)
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301)
    at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:64)
    at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:291)
    at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:245)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

How do I tell docker to use the IP address of the container or the name of the container as it starts and bind it to LISTENER_INTERNAL?

CodePudding user response：

You don't need an IP address of the container.

You've either not used --link (which is deprecated) or not used --network on a shared bridge.

If you want to make Kafka listen on all interfaces, you'd set listeners=<PROTOCOL>://0.0.0.0:<port>, but by default, it should bind to its hostname, and that won't solve DNS errors for external clients