I am creating logstash pipeline where I am giving log file as an input and reading those logs on elasticsearch. I want to add geoip filter in my logstash pipeline configuration but when I am adding it's failing and shutting down.

Here is an errors:

[2022-03-17T12:41:05,243][WARN ][logstash.outputs.elasticsearch][main] 
Elasticsearch Output configured with `ecs_compatibility => v8`, which 
resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common 
Schema. Once ECS v8 and an updated release of this plugin are publicly 
available, you will need to update this plugin to resolve this warning.
[2022-03-17T12:41:05,293][ERROR][logstash.javapipeline    ][main] 
Pipeline error {:pipeline_id=>"main", :exception=># 
<LogStash::ConfigurationError: GeoIP Filter in ECS-Compatiblity mode 
requires a `target` when `source` is not an `ip` sub-field, eg. [client] 
[ip]>, :backtrace=>["D:/logstash- 
8.1.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-7.2.11- 
java/lib/logstash/filters/geoip.rb:143:in `auto_target_from_source!'", 
"D:/logstash-8.1.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip- 
7.2.11-java/lib/logstash/filters/geoip.rb:133:in `setup_target_field'", 
"D:/logstash-8.1.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip- 
7.2.11-java/lib/logstash/filters/geoip.rb:108:in `register'", 
"org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in 
`register'", "D:/logstash-8.1.0/logstash- 
core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", 
"org/jruby/RubyArray.java:1821:in `each'", "D:/logstash-8.1.0/logstash- 
core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", 
"D:/logstash-8.1.0/logstash-core/lib/logstash/java_pipeline.rb:590:in 
`maybe_setup_out_plugins'", "D:/logstash-8.1.0/logstash- 
core/lib/logstash/java_pipeline.rb:244:in `start_workers'", 
"D:/logstash- 
8.1.0/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'", 
"D:/logstash-8.1.0/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["D:/logstash-8.1.0/my-logstash.conf"], :thread=>"#<Thread:0x6ea94258 run>"}
[2022-03-17T12:41:05,314][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-03-17T12:41:05,357][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-03-17T12:41:05,390][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2022-03-17T12:41:05,499][DEBUG][logstash.instrument.periodicpoller.os] Stopping
[2022-03-17T12:41:05,523][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping
[2022-03-17T12:41:05,525][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping
[2022-03-17T12:41:05,532][DEBUG] 
[logstash.instrument.periodicpoller.deadletterqueue] Stopping
[2022-03-17T12:41:05,556][DEBUG][logstash.agent           ] Shutting 
down all pipelines {:pipelines_count=>0}

When I am using below configuration without filter, then it's working fine:

input {
 file {
  path => "D:/nest/es-logging-example/log/info/*.log"
  start_position => beginning
  sincedb_path => "NULL"
 }
}

output {
 elasticsearch {
    hosts => "localhost:9200"
    index => "myapplogs"
 }
 stdout{}
}

But on adding filter in configuration file then it's failing and shutting down:

input { 
 file {
  path => "D:/nest/es-logging-example/log/info/*.log"
  start_position => beginning
  sincedb_path => "NULL"
  }
}
filter {
 geoip {
    source => "clientip"
 }
}
output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "myapplogs"
  }
  stdout{}
}

What I am doing wrong in 2nd configuration?

CodePudding user response：

What the error states is this

GeoIP Filter in ECS-Compatiblity mode requires a target when source is not an ip sub-field. You're simply missing an explicit target field

So your filter should look like this:

filter {
 geoip {
    source => "clientip"
    target => "clientgeo"
 }
}