Spring Boot app is not able to publish events to Kafka in Docker-CodePudding

Hello fellow programmers!

I am learning Docker and Kafka, and would be very grateful for any help. The problem is that the application is able to connect to Kafka, but not to publish any events. Error:

org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for wages-local-0:120001 ms has passed since batch creation

This problem arises when Spring Boot app in one Docker container tries to publish to Kafka in other container:

version: '3'

services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    hostname: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INTERNAL://kafka:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
    depends_on:
      - zookeeper
  publisher_app:
    build:
      context: ./kafka-publisher
    ports:
      - 8080:8080
    depends_on:
      - "kafka"
    environment:
      kafka.wages-topic.bootstrap-address: kafka:9092
    links:
      - kafka:kafka

Here is the link to source code: https://github.com/aleksei17/springboot-rest-kafka-mysql/blob/master/docker-compose.yml

CodePudding user response：

Your config is hard-coded to use localhost:9092

You want to externalize the config like so (remove your kafka.wages-topic section since bootstrap servers are not a topic-specific setting)

spring:
  kafka:
    bootstrap-servers: ${BOOTSTRAP_SERVERS}

Then in Compose,

environment:
  BOOTSTRAP_SERVERS: kafka:9092

To configure topics, see https://docs.spring.io/spring-kafka/reference/html/#configuring-topics

CodePudding user response：

Well, it seems that I needed to delete containers after usage. When I did so, I got an error when starting Kafka container like: unknown listener PLAINTEXT. So I changed PLAINTEXT to INTERNAL, and it worked! Here is the final version of docker-compose for Kafka:

  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    hostname: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INTERNAL://kafka:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
    depends_on:
      - zookeeper

It would be great if it worked both for Dockerized apps and for apps run on the same host outside Docker, but this is a different story. Played with this tutorial, but no success so far: https://www.baeldung.com/kafka-docker-connection