Logstash - splitting the log into a csv file-CodePudding

I want to use logstash to separate the appropriate logs by a constant value appearing in these logs, and then divide the log into pieces after the separator ("|") and put it into a csv file with headers. The logs I'm looking for are recognized by the constant (WID2). I also noticed that the message pulled out by GREEDYDATA gets cut off after about 85 characters

Example log: 2022-01-02 10:32:30,0000001 | WID2 | 3313141414 | Request | STEP_1 | OK | Message

And i want from this logs create csv file with headers: TIMESTAMP, VALUE, MESSAGE_TYPE, STEP, STATUS, MESSAGE. I do not want to save a constant value (WID2) in the csv file, it only serves to find my logs among others.

I wrote it but it doesn't work:

input {
 file {
  path => ["path"]
  start_position => "beginning"
  sincedb_path => "path"
 }
}

filter {
 grok {
  match => {
   "message" => "%{GREEDYDATA:SYSLOGMESSAGE}"
    }
   }
 if ([SYSLOGMESSAGE] !~ "WID2"){
  drop {}
 }
 if([SYSLOGMESSAGE] =~ 'WID2") {
  csv {
   separator => "|"
   columns => ["TIMESTAMP", "VALUE", "MESSAGE_TYPE", "STEP", "STATUS", "MESSAGE"]
  }
 }
}

output{
 file {
  path => ["path.csv"]
 }
}

CodePudding user response：

If your log messages have this format:

2022-01-02 10:32:30,0000001 | WID2 | 3313141414 | Request | STEP_1 | OK | Message

And you want to parse every message that has WID2 on it, the following filter will work.

filter {
    if "WID2" in [message] {
        csv {
            separator => "|"
            columns => ["TIMESTAMP", "[@metadata][wid2]", "VALUE", "MESSAGE_TYPE", "STEP", "STATUS", "MESSAGE"]
        }
    } else {
        drop {}
    }
}

The if conditional will test if WID2 is present in the message, if it is true, it will use the csv filter to parse it, since the second column of your csv is the value WID2 and you do not want to save it, you can store its value in the field [@metadata][wid2], this metadata field will not be present in the output block.

If the string WID2 is not present in the message field, the event is dropped.