I am new to kafka, i tried to setup my first cluster on a vps with Docker-compose. But i still cannot access it from my local pc ( outside the host ).

here is my docker compose

version: '2'
    services:
      zookeeper-1:
        image: confluentinc/cp-zookeeper:latest
        environment:
          ZOOKEEPER_CLIENT_PORT: 2181
          ZOOKEEPER_TICK_TIME: 2000
        
        ports:
          - 22181:2181

      zookeeper-2:
        image: confluentinc/cp-zookeeper:latest
        environment:
          ZOOKEEPER_CLIENT_PORT: 2181
          ZOOKEEPER_TICK_TIME: 2000
        ports:
          - 32181:2181

      kafka-1:
        image: confluentinc/cp-kafka:latest
        depends_on:
          - zookeeper-1
          - zookeeper-2

        ports:
          - 29092:29092
        environment:
          KAFKA_BROKER_ID: 1
          KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181
          KAFKA_LISTENERS: EXTERNAL_SAME_HOST://:29092,EXTERNAL_DIFFERENT_HOST://:29093,INTERNAL://:9092
          KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL_SAME_HOST://localhost:29092,EXTERNAL_DIFFERENT_HOST://XXX.XXX.XXX.XXX:29093
          KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT,EXTERNAL_DIFFERENT_HOST:PLAINTEXT
          KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL      
          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      kafka-2:
        image: confluentinc/cp-kafka:latest
        depends_on:
          - zookeeper-1
          - zookeeper-2
        ports:
          - 39092:39092
        environment:
          KAFKA_BROKER_ID: 2
          KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:2181,zookeeper-2:2181
          KAFKA_LISTENERS: EXTERNAL_SAME_HOST://:39092,EXTERNAL_DIFFERENT_HOST://:39093,INTERNAL://:9093
          KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9093,EXTERNAL_SAME_HOST://localhost:39092,EXTERNAL_DIFFERENT_HOST://XXX.XXX.XXX.XXX:39093
          KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT,EXTERNAL_DIFFERENT_HOST:PLAINTEXT
          KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

I searched in the logs and i found that there is no available brokers ( always 0 ) because the server couldn't connect to "kafka:9092" AND Zookeeper keep failing to connect to the brokers.

[2022-04-13 14:56:59,422] WARN Session 0x0 for sever My-vps-URL/XXX.XXX.XXX.XXX:2181, Closing socket connection. Attempting reconnect except 
it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not 
heard from server in 30006ms for session id 0x0
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1258)
KeeperErrorCode = ConnectionLoss for /brokers/ids

How can i fix this ?

Note that i tried a similar config with a different docker image ( bitnami's ), and with different cluster config ( 1 zookeeper 1 broker ) and it still doesn't work.

CodePudding user response：

You have a Zookeeper error because you are running an even number of them. The number of Zookeepers doesn't need to match the number of brokers, and it should be an odd number of them, only, up to 7 max. You also shouldn't need ports on the Zookeeper servers.

For your Kafka Connection,

1. KAFKA_LISTENERS needs to include the IP of 0.0.0.0 to allow for the server to bind on all interfaces
2. You need to expose port 29093 and 39093 since those are your "different host" settings. You currently only have ports to connect from the same machine.
3. your clients need to connect to the EXTERNAL_DIFFERENT_HOST address you've set, not kafka:9092

Further reading - Connect to Kafka running in Docker

tried a similar config with a different docker image ( bitnami's )

That image has different variables, but same basic answer as above.

different cluster config ( 1 zookeeper 1 broker )

There is little benefit of running multiple of each of those on the same machine, so I suggest trying to get that configuration working, first.