Trying to drop logs the end with certain string logstash-CodePudding

I'm trying to drop the logs for 200 codes in response to Prometheus scraping. In Kibana this is the message field:

November 17th 2021, 12:37:01.769    10.128.8.31 - - [17/Nov/2021:12:37:01  0000] "GET /metrics HTTP/1.1" 200 36881 "-" "Prometheus/2.25.0"

I've added the following to the filter in logstash config:

if [message] =~ /.*Prometheus\/2.25.0$/   {  
    drop { }  
  }

But the logs are still coming through, I've tried many variations but nothing seems to work so I'm unsure what I'm missing?

Thanks

CodePudding user response：

Since you're ingesting Apache logs, you can try to parse the line using a pre-defined grok pattern and then simply drop the event based on the user-agent.

Grokking the Apache log you've shared using the COMBINEDAPACHELOG pattern (more patterns can be found here) would parse the message field as follows:

{
  "clientip": [
    [
      "10.128.8.31"
    ]
  ],
  "ident": [
    [
      "-"
    ]
  ],
  "auth": [
    [
      "-"
    ]
  ],
  "timestamp": [
    [
      "17/Nov/2021:12:37:01  0000"
    ]
  ],
  "verb": [
    [
      "GET"
    ]
  ],
  "request": [
    [
      "/metrics"
    ]
  ],
  "httpversion": [
    [
      "1.1"
    ]
  ],
  "rawrequest": [
    [
      null
    ]
  ],
  "response": [
    [
      "200"
    ]
  ],
  "bytes": [
    [
      "36881"
    ]
  ],
  "referrer": [
    [
      ""-""
    ]
  ],
  "agent": [
    [
      ""Prometheus/2.25.0""
    ]
  ]
}

So now all you have to do is to drop the event based on the value of the agent field:

filter {
  # first grok the Apache log
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  # then drop if you want to ignore a given user-agent 
  if [agent] == "Prometheus\/2.25.0" {
    drop {}
  }
}

CodePudding user response：

Putting the check to drop inside the if statement where I was checking the kubernetes namespace works i.e

if "example-namespace" in [kubernetes][namespace] {
mutate { add_field => { "proj_index" => "example-namespace"} }                    json {
  source => "message"
} 
if [message] =~ /.*Prometheus\/2.25.0"$/ {
drop { }
  }
 }
}

It didn't work putting the drop conditional inside the filter but outside any particular namespace so it would apply to all of them for some reason.