I have created a multi-container app, and only one of the containers cannot connect to Kafka (container named "backend" raises error "kafka.errors.NoBrokersAvailable: NoBrokersAvailable"). The most strange is that container named "apigateway" that uses the same variables to connect Kafka works fine. The difference is that I use Flask with kafka-python lib inside of "apigateway" and clear Python with kafka-python lib inside of "backend".

However, when I run "backend" container outside of docker-compose using "docker run" and not connecting to docker bridge but connecting to containers from the host machine everything works fine. The command I use below:

docker run --env KAFKA_HOST=localhost --env KAFKA_PORT=9092 --env DATA_REQUEST_TOPIC=data-request-topic --env DB_TYPE='mongodb' --env DB_USERNAME='root' --env DB_PASSWORD='root' --env DB_HOST=localhost --env DB_PORT=27017 --env DB_NAME='backend_microservice_database' --env DB_COLLECTION='ms_collection' --net=host -it backend_microservice:latest

My docker-compose.yml below:

version: '2.1'
services:

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

  # reachable on 9092 from the host and on 29092 from inside docker compose
  kafka:
    image: confluentinc/cp-kafka:6.2.4
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
    expose:
      - '29092'
    environment:
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: '1'
      KAFKA_MIN_INSYNC_REPLICAS: '1'

  init-kafka:
    image: confluentinc/cp-kafka:6.2.4
    depends_on:
      - kafka
    entrypoint: [ '/bin/sh', '-c' ]
    command: |
      "
      # blocks until kafka is reachable
      kafka-topics --bootstrap-server kafka:29092 --list

      echo -e 'Creating kafka topics'
      kafka-topics --bootstrap-server kafka:29092 --create --if-not-exists --topic data-request-topic --replication-factor 1 --partitions 1
      kafka-topics --bootstrap-server kafka:29092 --create --if-not-exists --topic data-response-topic --replication-factor 1 --partitions 1
      kafka-topics --bootstrap-server kafka:29092 --create --if-not-exists --topic order_topic --replication-factor 1 --partitions 1

      echo -e 'Successfully created the following topics:'
      kafka-topics --bootstrap-server kafka:29092 --list"
      
  apigateway:
    image: diploma_gateway:latest
    depends_on:
      - kafka
    ports:
      - "5000:5000"
    expose:
      - '5000'
    environment:
      KAFKA_HOST: kafka
      KAFKA_PORT: 29092
      DATA_REQUEST_TOPIC: data-request-topic
      ALLOWED_OUTER_HOSTS: 0.0.0.0

  mongodb:
    image: mongo:5.0.8
    ports:
      - 27017:27017
    expose: 
      - 27017
    volumes:
      - ./database_storage:/data/db
    environment:
      - MONGO_INITDB_DATABASE=backend_microservice_database
      - MONGO_INITDB_ROOT_USERNAME=root
      - MONGO_INITDB_ROOT_PASSWORD=root
      
  backend:
    image: backend_microservice:latest
    restart: always
    network_mode: "bridge"
    depends_on:
      - mongodb
      - kafka
    environment:
      KAFKA_HOST: kafka
      KAFKA_PORT: 29092
      DATA_REQUEST_TOPIC: data-request-topic
      DB_TYPE: mongodb
      DB_USERNAME: root
      DB_PASSWORD: root
      DB_HOST: mongodb
      DB_PORT: 27017
      DB_NAME: backend_microservice_database
      DB_COLLECTION: ms_collection

could anyone please suggest what am I doing wrong?

CodePudding user response：

I can't really tell you what's wrong but here some thoughts:

depends_on just expresses dependencies between services and the start/stop-order. But compose is not waiting for the service being ready. There are 2 possibilities to handle that:

1. Make sure service stops with a failure code and restart: always, it has to be sure that it will not hang somewhere waiting.
2. Perform a Healthcheck in compose.

No need to expose ports in compose, normally they have to be defined in Dockerfile

Remove network_mode: "bridge", if you state nothing at all, compose will create a bridge network by default and all services will join it.

Kafka needs to know the allowed listeners:

KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,PLAINTEXT_HOST://127.0.0.1:9092

CodePudding user response：

A big thank all of you guys who took a part in helping me, I've removed network_mode as @OneCricketeer and @araisch told me and decided to remove all old containers and after that it all started to work. Probably there were some collisions because a lot of containers with same tags that were being rebuilt and restarted again and again.