I use filebeat to fetch log files into my logstash and then filter unnecessary fields. Everything works fine and I output these into elasticsearch but there is a field which I use for elasticsearch index name, I define this variable in my grok match but I couldn't find a way to remove that variable once it serves its purpose. I'll share my logstash config below

input {
    beats {
            port => "5044"
        }
}
filter {
    grok {
         match => { "[log][file][path]" => ".*(\\|\/)(?<myIndex>.*)(\\|\/).*.*(\\|\/).*(\\|\/).*(\\|\/).*(\\|\/)" }
    }
    json {
        source => message
    }
    mutate {            
        remove_field => ["agent"] 
        remove_field => ["input"] 
        remove_field => ["@metadata"]
        remove_field => ["log"]
        remove_field => ["tags"]    
        remove_field => ["host"]
        remove_field => ["@version"]
        remove_field => ["message"]
        remove_field => ["event"]
        remove_field => ["ecs"]
    }
    date {
        match => ["t","yyyy-MM-dd HH:mm:ss.SSS"]
            remove_field => ["t"] 
    }
    mutate {
        rename => ["l","log_level"]
        rename => ["mt","msg_template"]
        rename => ["p","log_props"]
    }
}
output {
   elasticsearch {
        hosts => [ "localhost:9222" ]
        index => "%{myIndex}"
   }
    stdout { codec => rubydebug { metadata => true } }
}

I just want to remove the "myIndex" field from my index. With this config file, I see this field in elasticsearch if possible I want to remove it. I've tried to remove it with other fields altogether but it gave an error. I guess it's because I removed it before logstash could give it to elasticsearch.

CodePudding user response：

Create the field under [@metadata]. Those fields are available to use in logstash but are ignored by outputs unless they use a rubydebug codec.

Adjust your grok filter

match => { "[log][file][path]" => ".*(\\|\/)(?<[@metadata][myIndex]>.*)(\\|\/).*.*(\\|\/).*(\\|\/).*(\\|\/).*(\\|\/)" }

Delete [@metadata] from the mutate remove_field and change the output configuration to have

index => "%{[@metadata][myIndex]}"