I am using AWS Opensearch to retrieve the logs from all my Kubernetes applications. I have the following pods: Kube-proxy, Fluent-bit, aws-node, aws-load-balancer-controller, and all my apps (around 10).

While fluent-bit successfully send all the logs from Kube-proxy, Fluent-bit, aws-node and aws-load-balancer-controller, none of the logs from my applications are sent. My applications had DEBUG, INFO, ERROR logs, and none are sent by fluent bit.

Here is my fluent bit configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: my-namespace
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     50MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            my-host.es.amazonaws.com
        Port            443
        TLS             On
        AWS_Auth        On
        AWS_Region      ap-southeast-1
        Retry_Limit     6

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S )(?:  (?<path>[^\"]*?)(?:  \S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S )(?:  (?<path>[^ ]*)  \S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S )(?:  (?<path>[^\"]*?)(?:  \S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9] )\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9] )\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

I followed this documentation: https://www.eksworkshop.com/intermediate/230_logging/deploy/

Thanks a lot for the help.

CodePudding user response：

have you seen this article from official side? Pay attention on Log files overview section.

When deploying Fluent Bit to Kubernetes, there are three log files that you need to pay attention to. C:\k\kubelet.err.log

Also you can find Fluent GitHub Community and create an issue there to have better support from its contributors

There is a Slack channel for Fluent

CodePudding user response：

Finally, I did two things that solved my issue:

1. Modified this configuration:

# before
 output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            search-blacaz-logs-szzq6vokwwm4y5fkfwyngjwjxq.ap-southeast-1.es.amazonaws.com
        Port            443
        TLS             On
        AWS_Auth        On
        AWS_Region      ap-southeast-1
        Retry_Limit     6

# after
 output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            search-blacaz-logs-szzq6vokwwm4y5fkfwyngjwjxq.ap-southeast-1.es.amazonaws.com
        Port            443
        TLS             On
        AWS_Auth        On
        Replace_Dots    On // added this
        AWS_Region      ap-southeast-1
        Retry_Limit     6

Then, I had to delete the fluent-bit Elastic search index, and re-create it. Indeed, the index was probably not well suited for my JAVA logs at first, and adjusted to it after re-creation.