Grok Pattern to Parse Multiple loglines-CodePudding

I have 3 lines of logs with different structure. I am trying to construct a grok pattern to filter the logs.

[2022-10-04 21:45:27,444: INFO/MainProcess] Events of group {task} enabled by remote
[2022-10-04 21:43:06,521: ERROR/MainProcess] consumer: Cannot connect to redis://10.0.13.57:6379/0: Error 111 connecting to 10.0.13.34:6379. Connection refused..
[2022-10-04 21:45:22  0000] [3094] [INFO] Listening at: http://0.0.0.0:8793 (3094)

I am expecting:

timestamp: loglevel: message:

The grok pattern I have doesn't match anything: \[%{TIMESTAMP_ISO8601:timestamp}\]\:%{LOGLEVEL:loglevel}%{WORD: class} %{SPACE}%{GREEDYDATA:logMessage}

CodePudding user response：

You need to have two grok pattern for separate logs.

[2022-10-04 21:45:27,444: INFO/MainProcess] Events of group {task} enabled by remote
[2022-10-04 21:43:06,521: ERROR/MainProcess] consumer: Cannot connect to redis://10.0.13.57:6379/0: Error 111 connecting to 10.0.13.34:6379. Connection refused..

The grok pattern for the above two logs:

%{DATESTAMP:timestamp}\: %{LOGLEVEL:loglevel}\/%{DATA:data}\] %{GREEDYDATA:message}

Output:

[2022-10-04 21:45:22  0000] [3094] [INFO] Listening at: http://0.0.0.0:8793 (3094)

The grok pattern for the above log:

\[%{TIMESTAMP_ISO8601:timestamp} \ %{DATA:data}\] \[%{LOGLEVEL:loglevel}\] %{GREEDYDATA:message}

Output:

Also, you can make use of the Drop Filter of the logstash to drop the field data generated [see the output screenshot below] after parsing your logs using the above GROK pattern.