Not working logstash multiple kafka topic-CodePudding

logstash version is logstash-7.16.3

here is conf file

input {
  kafka {
    bootstrap_servers => "abc-private-vip:9092"
    topics => ["label-result","text-result"]
    group_id => "result-group"
    consumer_threads => 2
    decorate_events => true
  }
}

output {
  if [@metadata][kafka][topic] == "label-result" {
    elasticsearch {
      hosts => ["abc-private-vip:9200"]
      index => "label-result-%{ YYYYMMdd}"
    }
  }

  if [@metadata][kafka][topic] == "text-result" {
    elasticsearch {
      hosts => ["tcore-private-vip:9200"]
      index => "text-result-%{ YYYYMMdd}"
    }
  }
}

If "if [@metadata][kafka][topic] == "text-result" {...}" is removed, it works well.

What's wrong? so simple..

CodePudding user response：

The decorate_events parameter is not of boolean type but string type and takes different values (true and false are deprecated):

none (= deprecated "false" value)
basic (= deprecated "true" value)
extended

So you should change your input configuration to this instead

decorate_events => "basic"

Also you can add the following output to see if the @metadata field is correctly being added to your events

output {
    stdout {codec => rubydebug}
}

CodePudding user response：

so i fixed it

input {
  kafka {
    bootstrap_servers => "tcore-private-vip:9092"
    topics => ["label-result","text-result"]
    group_id => "result-group"
    consumer_threads => 3
    decorate_events => "basic"
  }
}

filter {
  mutate {
    add_field => {
      "kafka-topic" => "%{[@metadata][kafka][topic]}"
    }
  }
}

output {
  elasticsearch {
    hosts => ["tcore-private-vip:9200"]
    index => "%{kafka-topic}-%{ YYYYMMdd}"
  }
}


"@timestamp" => 2022-03-14T08:38:45.250Z,
        "message" => "",
       "@version" => "1",
    "kafka-topic" => "label-result"

metadata is exist but not pushed to elasticsearch