Fluentd - Kubernetes - ParserError error="pattern not matched with data"-CodePudding

I'm trying to redirect Kubernetes logs from containers to OpenSearch. But there is always some error with the date. What am I doing wrong?

Docker logs example:

{"log":"time=\"2022-04-01T10:02:31Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-0d1744c2-ff8d-4a68-8a2c-fbfd16408975' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker2\n","stream":"stderr","time":"2022-04-01T10:02:31.230191143Z"}
{"log":"E0401 10:02:31.230146       1 backup_controller.go:153] longhorn-backup: fail to sync backup longhorn-system/backup-989764daba094e0d: could not find snapshot 'snapshot-0d1744c2-ff8d-4a68-8a2c-fbfd16408975' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\n","stream":"stderr","time":"2022-04-01T10:02:31.230214608Z"}
{"log":"time=\"2022-04-01T10:02:31Z\" level=warning msg=\"Dropping Longhorn backup longhorn-system/backup-989764daba094e0d out of the queue\" controller=longhorn-backup error=\"longhorn-backup: fail to sync backup longhorn-system/backup-989764daba094e0d: could not find snapshot 'snapshot-0d1744c2-ff8d-4a68-8a2c-fbfd16408975' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker2\n","stream":"stderr","time":"2022-04-01T10:02:31.230218285Z"}

Fluentd Output:

fluentd/fluentd-x7pgc[fluentd]: 2022-04-01 09:54:33  0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n'" location=nil tag="kubernetes.var.log.containers.longhorn-manager-5lmdf_longhorn-system_longhorn-manager-5f6bc9870a9efe75670274d177d0bf17dee0dd995a433343432b3155af946823.log" time=2022-04-01 09:54:33.524389862  0000 record={"log"=>"time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n", "stream"=>"stderr"}
fluentd/fluentd-x7pgc[fluentd]: 2022-04-01 09:54:33  0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Error syncing Longhorn backup longhorn-system/backup-c5104cf80da04be6\" controller=longhorn-backup error=\"longhorn-backup: fail to sync backup longhorn-system/backup-c5104cf80da04be6: could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n'" location=nil tag="kubernetes.var.log.containers.longhorn-manager-5lmdf_longhorn-system_longhorn-manager-5f6bc9870a9efe75670274d177d0bf17dee0dd995a433343432b3155af946823.log" time=2022-04-01 09:54:33.524404952  0000 record={"log"=>"time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Error syncing Longhorn backup longhorn-system/backup-c5104cf80da04be6\" controller=longhorn-backup error=\"longhorn-backup: fail to sync backup longhorn-system/backup-c5104cf80da04be6: could not find snapshot 'snapshot-15dfec02-b8c4-40db-a7ed-bf84429ac220' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n", "stream"=>"stderr"}
fluentd/fluentd-x7pgc[fluentd]: 2022-04-01 09:54:33  0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-9d5705bf-26fc-49e5-a771-cf9352049c04' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n'" location=nil tag="kubernetes.var.log.containers.longhorn-manager-5lmdf_longhorn-system_longhorn-manager-5f6bc9870a9efe75670274d177d0bf17dee0dd995a433343432b3155af946823.log" time=2022-04-01 09:54:33.538539106  0000 record={"log"=>"time=\"2022-04-01T09:54:33Z\" level=warning msg=\"Cannot take snapshot backup\" controller=longhorn-backup error=\"could not find snapshot 'snapshot-9d5705bf-26fc-49e5-a771-cf9352049c04' to backup, volume 'pvc-03557105-c20d-4fbe-8d0d-8a0b4ac16f6d'\" node=k8s-worker1\n", "stream"=>"stderr"}

Config:

<source>
  @type tail
  @id tail_all_container_logs
  @label @FLUENTD.OPENSEARCH
  path /var/log/containers/longhorn*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  exclude_path "#{ENV['FLUENT_ALL_CONTAINERS_TAIL_EXCLUDE_PATHS']}"
  <parse>
      @type json
  </parse>
</source>

<filter kubernetes.**>
    @type parser
    key_name log
    <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%N%z
        timezone  00:00
    </parse>
</filter>

CodePudding user response：

The date/time from your logs is 2022-04-01T09:54:33Z note no milliseconds.

While in your config has time_format %Y-%m-%dT%H:%M:%S.%N%z

%N - Fractional seconds digits, default is 9 digits (nanosecond).

Try to remove the .%N part to match with your time format from logs. Which would be:

time_format %Y-%m-%dT%H:%M:%S%z

For more information about the time format syntax kindly refer to this page.

CodePudding user response：

@piotr-malec - Thanks for the answer. I've tried many different date/time formats but it's always the same.

<filter kubernetes.**>
    @type parser
    key_name log
    <parse>
        @type json
        time_format %Y-%m-%dT%H:%M:%S.%N%z
        #time_format %Y-%m-%dT%H:%M:%S.%z
        #time_format %Y-%m-%dT%H:%M:%S%z
        #time_format %Y-%m-%dT%H:%M:%Sz
        #time_format %Y-%m-%dT%H:%M:%SZ
        #time_format %Y-%m-%dT%H:%M:%S%Z
        #time_format %Y-%m-%dT%H:%M:%S.%z
        timezone  00:00
    </parse>
</filter>

Now I have a similar case with a different log.

fluentd/fluentd-9gfjs[fluentd]: 2022-04-05 08:46:58  0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data '[2022-04-05T08:46:58,305][INFO ][o.o.j.s.JobSweeper       ] [opensearch-cluster-master-0] Running full sweep\n'" location=nil tag="opensearch-master" time=1970-01-01 00:33:42.306044198  0000 record={"log"=>"[2022-04-05T08:46:58,305][INFO ][o.o.j.s.JobSweeper       ] [opensearch-cluster-master-0] Running full sweep\n", "stream"=>"stdout"}
fluentd/fluentd-9gfjs[fluentd]: 2022-04-05 08:51:58  0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data '[2022-04-05T08:51:58,306][INFO ][o.o.j.s.JobSweeper       ] [opensearch-cluster-master-0] Running full sweep\n'" location=nil tag="opensearch-master" time=1970-01-01 00:33:42.306370871  0000 record={"log"=>"[2022-04-05T08:51:58,306][INFO ][o.o.j.s.JobSweeper       ] [opensearch-cluster-master-0] Running full sweep\n", "stream"=>"stdout"}

I have tried the following formats. Each of them does not work.

<filter opensearch-master>
    @type parser
    key_name log
    <parse>
        @type json
        time_key @timestamp
        time_format %Y-%m-%dT%H:%M:%S,%3N
        #time_format %Y-%m-%dT%H:%M:%S,%L
        timezone  02:00
    </parse>
</filter>

Ruby Documentation: https://docs.ruby-lang.org/en/2.4.0/Time.html#method-i-strftime