I am trying to run Spring application with logstash, Elastic Search, Kafka and Kibana.

[main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://kafka:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://kafka:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

The above error is being repeated under logstash container

docker-compose.yml

version: '3.7'
services:

  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    #restart: always
    networks:
      - tweetapp-network
  
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    #restart: always
    ports: 
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "tweetapp-logs:1:1, Tweets:1:1"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - tweetapp-network
  
  mongodb:
    image: mongo
    container_name: mongodb
   # restart: always
    ports:
      - "27017:27017"
#    volumes:
#      - mongodb-volume:/data/db
    networks:
      - tweetapp-network
  
  springboot:
    image: tweetapp
    #restart: always
    ports:
      - "8080:8080"
    depends_on:
      - mongodb
      - kafka
      - elasticsearch
      - logstash
      - kibana
    networks:
      - tweetapp-network
  
  logstash:
    image: logstash:7.7.0
    container_name: logstash 
    hostname: logstash
    ports:
      - "9600:9600"
    volumes:
      - .\logstash:/usr/share/logstash/pipeline/
    links:
      - elasticsearch:elasticsearch
    depends_on:
      - elasticsearch
    networks:
      - tweetapp-network
  
  elasticsearch:
    image: elasticsearch:7.7.0
    container_name: elasticsearch
    hostname: elasticsearch
    ports:
      - "9200:9200"
    environment:
      - discovery.type=single-node
    networks:
      - tweetapp-network
      
  kibana:
    image: kibana:7.7.0
    container_name: kibana
    hostname: kibana
    ports:
      - "5601:5601"
    links:
      - elasticsearch:elasticsearch
    depends_on:
      - elasticsearch
    networks:
      - tweetapp-network
     
# Networks to be created to facilitate communication between containers
networks:
  tweetapp-network:

I made sure that Elastic Search is working http://localhost:9200/ , for this URL I get JSON output.

logstash.config

input {
    kafka {
            bootstrap_servers => "kafka:9092"
            topics => ["tweetapp-logs"]
    }
}
filter {
grok {
    match => [ "message", "%{GREEDYDATA}" ]
  } 
}
output {
   elasticsearch {
      hosts => ["kafka:9200"]
      index => "tweetapp"
      workers => 1
    }
}

I am new to docker-compose, Elastic search and Kafka. Any help will be appreciated.

CodePudding user response：

Seems like you are unlucky.

Replace Elasticsearch instead of kafka in host name

Try this:

input { kafka { bootstrap_servers => "kafka:9092" topics => ["tweetapp-logs"] } } filter { grok { match => [ "message", "%{GREEDYDATA}" ] } } output { elasticsearch { hosts => ["elasticsearch:9200"] index => "tweetapp" workers => 1 } }