Kafka cannot connect to zookeeper - connection refused-CodePudding

I have this setup that has been working fine since last year December which suddenly refused to work.

I have this docker-compose yaml file like this:

version: "3.8"

services:
  
  zookeeper1:
    image: debezium/zookeeper:1.8
    container_name: zookeeper1
    ports:
      - 2181:2181
    networks:
      - internalnet

  kafka1:
    image: debezium/kafka:1.8
    container_name: kafka1
    ports:
     - 9092:9092
    depends_on:
     - zookeeper1
    environment:
     - KAFKA_BROKER_ID=100
     - KAFKA_ZOOKEEPER_CONNECT=zookeeper1:2181
     - KAFKA_ADVERTISED_HOST_NAME=kafka1
     - KAFKA_LISTENERS=LISTENER_BOB://kafka1:29092,LISTENER_FRED://localhost:9092
     - KAFKA_ADVERTISED_LISTENERS=LISTENER_BOB://kafka1:29092,LISTENER_FRED://localhost:9092
     - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=LISTENER_BOB:PLAINTEXT,LISTENER_FRED:PLAINTEXT
     - KAFKA_INTER_BROKER_LISTENER_NAME=LISTENER_BOB
     - KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS=60000
    networks:
      - internalnet
    volumes:
      - ./kafka/kafka1/kafka_data:/kafka/data
      - ./kafka/kafka1/kafka_logs:/kafka/logs

networks:
  internalnet:
    driver: bridge

Zookeeper is running ok but kafka fails to run with the following log:

WARNING: Using default NODE_ID=1, which is valid only for non-clustered installations.
Starting in ZooKeeper mode using NODE_ID=1.
Using ZOOKEEPER_CONNECT=0.0.0.0:2181
Using configuration config/server.properties.
Using KAFKA_LISTENERS=LISTENER_BOB://kafka1:29092,LISTENER_FRED://localhost:9092 and KAFKA_ADVERTISED_LISTENERS=LISTENER_BOB://kafka1:29092,LISTENER_FRED://localhost:9092
2022-09-16 16:26:57,844 - INFO  [main:Log4jControllerRegistration$@31] - Registered kafka:type=kafka.Log4jController MBean
2022-09-16 16:26:58,521 - INFO  [main:X509Util@77] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2022-09-16 16:26:58,667 - INFO  [main:LoggingSignalHandler@72] - Registered signal handlers for TERM, INT, HUP
2022-09-16 16:26:58,674 - INFO  [main:Logging@66] - starting
2022-09-16 16:26:58,678 - INFO  [main:Logging@66] - Connecting to zookeeper on 0.0.0.0:2181
2022-09-16 16:26:58,719 - INFO  [main:Logging@66] - [ZooKeeperClient Kafka server] Initializing a new session to 0.0.0.0:2181.        
2022-09-16 16:26:58,733 - INFO  [main:Environment@98] - Client environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT
2022-09-16 16:26:58,734 - INFO  [main:Environment@98] - Client environment:host.name=44841d8b6caa
2022-09-16 16:26:58,734 - INFO  [main:Environment@98] - Client environment:java.version=11.0.14.1
2022-09-16 16:26:58,734 - INFO  [main:Environment@98] - Client environment:java.vendor=Red Hat, Inc.
2022-09-16 16:26:58,734 - INFO  [main:Environment@98] - Client environment:java.home=/usr/lib/jvm/java-11-openjdk-11.0.14.1.1-5.fc34.x86_64
2022-09-16 16:26:58,735 - INFO  [main:Environment@98] - Client environment:java.class.path=/kafka/libs/activation-1.1.1.jar:/kafka/libs/aopalliance-repackaged-2.6.1.jar:/kafka/libs/argparse4j-0.7.0.jar:/kafka/libs/audience-annotations-0.5.0.jar:/kafka/libs/commons-cli-1.4.jar:/kafka/libs/commons-lang3-3.8.1.jar:/kafka/libs/connect-api-3.0.0.jar:/kafka/libs/connect-basic-auth-extension-3.0.0.jar:/kafka/libs/connect-file-3.0.0.jar:/kafka/libs/connect-json-3.0.0.jar:/kafka/libs/connect-mirror-3.0.0.jar:/kafka/libs/connect-mirror-client-3.0.0.jar:/kafka/libs/connect-runtime-3.0.0.jar:/kafka/libs/connect-transforms-3.0.0.jar:/kafka/libs/hk2-api-2.6.1.jar:/kafka/libs/hk2-locator-2.6.1.jar:/kafka/libs/hk2-utils-2.6.1.jar:/kafka/libs/jackson-annotations-2.12.3.jar:/kafka/libs/jackson-core-2.12.3.jar:/kafka/libs/jackson-databind-2.12.3.jar:/kafka/libs/jackson-dataformat-csv-2.12.3.jar:/kafka/libs/jackson-datatype-jdk8-2.12.3.jar:/kafka/libs/jackson-jaxrs-base-2.12.3.jar:/kafka/libs/jackson-jaxrs-json-provider-2.12.3.jar:/kafka/libs/jackson-module-jaxb-annotations-2.12.3.jar:/kafka/libs/jackson-module-scala_2.12-2.12.3.jar:/kafka/libs/jakarta.activation-api-1.2.1.jar:/kafka/libs/jakarta.annotation-api-1.3.5.jar:/kafka/libs/jakarta.inject-2.6.1.jar:/kafka/libs/jakarta.validation-api-2.0.2.jar:/kafka/libs/jakarta.ws.rs-api-2.1.6.jar:/kafka/libs/jakarta.xml.bind-api-2.3.2.jar:/kafka/libs/javassist-3.27.0-GA.jar:/kafka/libs/javax.servlet-api-3.1.0.jar:/kafka/libs/javax.ws.rs-api-2.1.1.jar:/kafka/libs/jaxb-api-2.3.0.jar:/kafka/libs/jersey-client-2.34.jar:/kafka/libs/jersey-common-2.34.jar:/kafka/libs/jersey-container-servlet-2.34.jar:/kafka/libs/jersey-container-servlet-core-2.34.jar:/kafka/libs/jersey-hk2-2.34.jar:/kafka/libs/jersey-server-2.34.jar:/kafka/libs/jetty-client-9.4.43.v20210629.jar:/kafka/libs/jetty-continuation-9.4.43.v20210629.jar:/kafka/libs/jetty-http-9.4.43.v20210629.jar:/kafka/libs/jetty-io-9.4.43.v20210629.jar:/kafka/libs/jetty-security-9.4.43.v20210629.jar:/kafka/libs/jetty-server-9.4.43.v20210629.jar:/kafka/libs/jetty-servlet-9.4.43.v20210629.jar:/kafka/libs/jetty-servlets-9.4.43.v20210629.jar:/kafka/libs/jetty-util-9.4.43.v20210629.jar:/kafka/libs/jetty-util-ajax-9.4.43.v20210629.jar:/kafka/libs/jline-3.12.1.jar:/kafka/libs/jopt-simple-5.0.4.jar:/kafka/libs/kafka-clients-3.0.0.jar:/kafka/libs/kafka-log4j-appender-3.0.0.jar:/kafka/libs/kafka-metadata-3.0.0.jar:/kafka/libs/kafka-raft-3.0.0.jar:/kafka/libs/kafka-server-common-3.0.0.jar:/kafka/libs/kafka-shell-3.0.0.jar:/kafka/libs/kafka-storage-3.0.0.jar:/kafka/libs/kafka-storage-api-3.0.0.jar:/kafka/libs/kafka-streams-3.0.0.jar:/kafka/libs/kafka-streams-examples-3.0.0.jar:/kafka/libs/kafka-streams-scala_2.12-3.0.0.jar:/kafka/libs/kafka-streams-test-utils-3.0.0.jar:/kafka/libs/kafka-tools-3.0.0.jar:/kafka/libs/kafka_2.12-3.0.0.jar:/kafka/libs/log4j-1.2.17.jar:/kafka/libs/lz4-java-1.7.1.jar:/kafka/libs/maven-artifact-3.8.1.jar:/kafka/libs/metrics-core-2.2.0.jar:/kafka/libs/metrics-core-4.1.12.1.jar:/kafka/libs/netty-buffer-4.1.62.Final.jar:/kafka/libs/netty-codec-4.1.62.Final.jar:/kafka/libs/netty-common-4.1.62.Final.jar:/kafka/libs/netty-handler-4.1.62.Final.jar:/kafka/libs/netty-resolver-4.1.62.Final.jar:/kafka/libs/netty-transport-4.1.62.Final.jar:/kafka/libs/netty-transport-native-epoll-4.1.62.Final.jar:/kafka/libs/netty-transport-native-unix-common-4.1.62.Final.jar:/kafka/libs/osgi-resource-locator-1.0.3.jar:/kafka/libs/paranamer-2.8.jar:/kafka/libs/plexus-utils-3.2.1.jar:/kafka/libs/reflections-0.9.12.jar:/kafka/libs/rocksdbjni-6.19.3.jar:/kafka/libs/scala-collection-compat_2.12-2.4.4.jar:/kafka/libs/scala-java8-compat_2.12-1.0.0.jar:/kafka/libs/scala-library-2.12.14.jar:/kafka/libs/scala-logging_2.12-3.9.3.jar:/kafka/libs/scala-reflect-2.12.14.jar:/kafka/libs/slf4j-api-1.7.30.jar:/kafka/libs/slf4j-log4j12-1.7.30.jar:/kafka/libs/snappy-java-1.1.8.1.jar:/kafka/libs/trogdor-3.0.0.jar:/kafka/libs/zookeeper-3.6.3.jar:/kafka/libs/zookeeper-jute-3.6.3.jar:/kafka/libs/zstd-jni-1.5.0-2.jar     
2022-09-16 16:26:58,740 - INFO  [main:Environment@98] - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2022-09-16 16:26:58,745 - INFO  [main:Environment@98] - Client environment:java.io.tmpdir=/tmp
2022-09-16 16:26:58,745 - INFO  [main:Environment@98] - Client environment:java.compiler=<NA>
2022-09-16 16:26:58,745 - INFO  [main:Environment@98] - Client environment:os.name=Linux
2022-09-16 16:26:58,748 - INFO  [main:Environment@98] - Client environment:os.arch=amd64
2022-09-16 16:26:58,748 - INFO  [main:Environment@98] - Client environment:os.version=5.10.16.3-microsoft-standard-WSL2
2022-09-16 16:26:58,748 - INFO  [main:Environment@98] - Client environment:user.name=kafka
2022-09-16 16:26:58,748 - INFO  [main:Environment@98] - Client environment:user.home=/kafka
2022-09-16 16:26:58,749 - INFO  [main:Environment@98] - Client environment:user.dir=/kafka
2022-09-16 16:26:58,749 - INFO  [main:Environment@98] - Client environment:os.memory.free=975MB
2022-09-16 16:26:58,749 - INFO  [main:Environment@98] - Client environment:os.memory.max=1024MB
2022-09-16 16:26:58,749 - INFO  [main:Environment@98] - Client environment:os.memory.total=1024MB
2022-09-16 16:26:58,754 - INFO  [main:ZooKeeper@1006] - Initiating client connection, connectString=0.0.0.0:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@3fc79729
2022-09-16 16:26:58,782 - INFO  [main:ClientCnxnSocket@239] - jute.maxbuffer value is 4194304 Bytes
2022-09-16 16:26:58,797 - INFO  [main:ClientCnxn@1736] - zookeeper.request.timeout value is 0. feature enabled=false
2022-09-16 16:26:58,807 - INFO  [main:Logging@66] - [ZooKeeperClient Kafka server] Waiting until connected.
2022-09-16 16:26:58,840 - INFO  [main-SendThread(0.0.0.0:2181):ClientCnxn$SendThread@1181] - Opening socket connection to server 0.0.0.0/0.0.0.0:2181.
2022-09-16 16:26:58,842 - INFO  [main-SendThread(0.0.0.0:2181):ClientCnxn$SendThread@1183] - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2022-09-16 16:26:58,861 - WARN  [main-SendThread(0.0.0.0:2181):ClientCnxn$SendThread@1300] - Session 0x0 for sever 0.0.0.0/0.0.0.0:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)

Also got the following log from kafka server.log

2022-09-16 14:26:52,915 - INFO  [main:Log4jControllerRegistration$@31] - Registered kafka:type=kafka.Log4jController MBean
2022-09-16 14:26:54,942 - INFO  [main:Logging@66] - starting
2022-09-16 14:26:54,965 - INFO  [main:Logging@66] - Connecting to zookeeper on 0.0.0.0:2181
2022-09-16 14:26:55,082 - INFO  [main:Logging@66] - [ZooKeeperClient Kafka server] Initializing a new session to 0.0.0.0:2181.
2022-09-16 14:26:55,341 - INFO  [main:Logging@66] - [ZooKeeperClient Kafka server] Waiting until connected.
2022-09-16 14:27:55,367 - INFO  [main:Logging@66] - [ZooKeeperClient Kafka server] Closing.
2022-09-16 14:27:55,758 - INFO  [main:Logging@66] - [ZooKeeperClient Kafka server] Closed.
2022-09-16 14:27:55,797 - ERROR [main:MarkerIgnoringBase@159] - Fatal error during KafkaServer startup. Prepare to shutdown
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
    at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:254)
    at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:250)
    at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:108)
    at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1981)
    at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:457)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:196)
    at kafka.Kafka$.main(Kafka.scala:109)
    at kafka.Kafka.main(Kafka.scala)
2022-09-16 14:27:55,813 - INFO  [main:Logging@66] - shutting down
2022-09-16 14:27:55,858 - INFO  [main:Logging@66] - shut down completed
2022-09-16 14:27:55,861 - ERROR [main:MarkerIgnoringBase@143] - Exiting Kafka.
2022-09-16 14:27:55,864 - INFO  [kafka-shutdown-hook:Logging@66] - shutting down
2022-09-16 14:42:16,757 - INFO  [main:Log4jControllerRegistration$@31] - Registered kafka:type=kafka.Log4jController MBean
2022-09-16 14:42:18,622 - INFO  [main:Logging@66] - starting
2022-09-16 14:42:18,624 - INFO  [main:Logging@66] - Connecting to zookeeper on 0.0.0.0:2181
2022-09-16 14:42:18,656 - INFO  [main:Logging@66] - [ZooKeeperClient Kafka server] Initializing a new session to 0.0.0.0:2181.
2022-09-16 14:42:18,749 - INFO  [main:Logging@66] - [ZooKeeperClient Kafka server] Waiting until connected.
2022-09-16 14:43:18,769 - INFO  [main:Logging@66] - [ZooKeeperClient Kafka server] Closing.
2022-09-16 14:43:19,784 - INFO  [main:Logging@66] - [ZooKeeperClient Kafka server] Closed.
2022-09-16 14:43:19,796 - ERROR [main:MarkerIgnoringBase@159] - Fatal error during KafkaServer startup. Prepare to shutdown
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
    at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:254)
    at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:250)
    at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:108)
    at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1981)
    at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:457)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:196)
    at kafka.Kafka$.main(Kafka.scala:109)
    at kafka.Kafka.main(Kafka.scala)
2022-09-16 14:43:19,809 - INFO  [main:Logging@66] - shutting down
2022-09-16 14:43:19,858 - INFO  [main:Logging@66] - shut down completed
2022-09-16 14:43:19,870 - ERROR [main:MarkerIgnoringBase@143] - Exiting Kafka.
2022-09-16 14:43:19,876 - INFO  [kafka-shutdown-hook:Logging@66] - shutting down
2022-09-16 14:53:57,029 - INFO  [main:Log4jControllerRegistration$@31] - Registered kafka:type=kafka.Log4jController MBean
2022-09-16 14:53:59,011 - INFO  [main:Logging@66] - starting
2022-09-16 14:53:59,017 - INFO  [main:Logging@66] - Connecting to zookeeper on 0.0.0.0:2181
2022-09-16 14:53:59,115 - INFO  [main:Logging@66] - [ZooKeeperClient Kafka server] Initializing a new session to 0.0.0.0:2181.
2022-09-16 14:53:59,247 - INFO  [main:Logging@66] - [ZooKeeperClient Kafka server] Waiting until connected.
2022-09-16 14:54:59,256 - INFO  [main:Logging@66] - [ZooKeeperClient Kafka server] Closing.
2022-09-16 14:55:00,389 - INFO  [main:Logging@66] - [ZooKeeperClient Kafka server] Closed.
2022-09-16 14:55:00,397 - ERROR [main:MarkerIgnoringBase@159] - Fatal error during KafkaServer startup. Prepare to shutdown
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
    at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:254)
    at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:250)
    at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:108)
    at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1981)
    at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:457)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:196)
    at kafka.Kafka$.main(Kafka.scala:109)
    at kafka.Kafka.main(Kafka.scala)
2022-09-16 14:55:00,400 - INFO  [main:Logging@66] - shutting down
2022-09-16 14:55:00,491 - INFO  [main:Logging@66] - shut down completed
2022-09-16 14:55:00,525 - ERROR [main:MarkerIgnoringBase@143] - Exiting Kafka.
2022-09-16 14:55:00,529 - INFO  [kafka-shutdown-hook:Logging@66] - shutting down

I even went as far as uninstalling docker-desktop and re-installing, still the same issue.

Extra info: i am running docker-desktop (with wsl-2 on ubuntu distro) on windows 11.

CodePudding user response：

It's possible Debezium did an update which broke your setup, so I suggest you grab a latest compose file, many of which exist at

https://github.com/debezium/debezium-examples

Look at the logs,

Using ZOOKEEPER_CONNECT=0.0.0.0:2181

It's not using KAFKA_ZOOKEEPER_CONNECT ... Remove the KAFKA_ prefix to set the appropriate value, then your logs should say something like ZOOKEEPER_CONNECT=zookeeper1:2181