Folder deleted/not created inside the common dir mounted with emptyDir{} type on EKS Fargate pod-CodePudding

We are facing strange issue with EKS Fargate Pods. We want to push logs to cloudwatch with sidecar fluent-bit container and for that we are mounting the separately created /logs/boot and /logs/access folders on both the containers with emptyDir: {} type. But somehow the access folder is getting deleted. When we tested this setup in local docker it produced desired results and things were working fine but not when deployed in the EKS fargate. Below is our manifest files

Dockerfile

FROM anapsix/alpine-java:8u201b09_server-jre_nashorn

ARG LOG_DIR=/logs

# Install base packages
RUN apk update
RUN apk upgrade
# RUN apk add ca-certificates && update-ca-certificates

# Dynamically set the JAVA_HOME path
RUN export JAVA_HOME="$(dirname $(dirname $(readlink -f $(which java))))" && echo $JAVA_HOME

# Add Curl
RUN apk --no-cache add curl

RUN mkdir -p $LOG_DIR/boot $LOG_DIR/access
RUN chmod -R 0777 $LOG_DIR/*

# Add metadata to the image to describe which port the container is listening on at runtime.

# Change TimeZone
RUN apk add --update tzdata
ENV TZ="Asia/Kolkata"

# Clean APK cache
RUN rm -rf /var/cache/apk/*

# Setting JAVA HOME
ENV JAVA_HOME=/opt/jdk

# Copy all files and folders
COPY . .
RUN rm -rf /opt/jdk/jre/lib/security/cacerts
COPY cacerts /opt/jdk/jre/lib/security/cacerts
COPY standalone.xml /jboss-eap-6.4-integration/standalone/configuration/

# Set the working directory.
WORKDIR /jboss-eap-6.4-integration/bin

EXPOSE 8177

CMD ["./erctl"]

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vinintegrator
  namespace: eretail
  labels:
    app: vinintegrator
    pod: fargate
spec:
  selector:
    matchLabels:
      app: vinintegrator
      pod: fargate
  replicas: 2
  template:
    metadata:
      labels:
        app: vinintegrator
        pod: fargate
    spec:
      securityContext:
        fsGroup: 0
      serviceAccount: eretail
      containers:
      - name: vinintegrator
        imagePullPolicy: IfNotPresent
        image: 653580443710.dkr.ecr.ap-southeast-1.amazonaws.com/vinintegrator-service:latest
        resources:
          limits:
            memory: "7629Mi"
            cpu: "1.5"
          requests:
            memory: "5435Mi"
            cpu: "750m"
        ports:
        - containerPort: 8177
          protocol: TCP
        # securityContext:
          # runAsUser: 506
          # runAsGroup: 506
        volumeMounts:
          - mountPath: /jboss-eap-6.4-integration/bin
            name: bin
          - mountPath: /logs
            name: logs
      - name: fluent-bit
        image: 657281243710.dkr.ecr.ap-southeast-1.amazonaws.com/fluent-bit:latest
        imagePullPolicy: IfNotPresent
        env:
          - name: HOST_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 200m
            memory: 100Mi
        volumeMounts:
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
        - name: logs
          mountPath: /logs
          readOnly: true
      volumes:
        - name: fluent-bit-config
          configMap:
            name: fluent-bit-config
        - name: logs
          emptyDir: {}
        - name: bin
          persistentVolumeClaim:
            claimName: vinintegrator-pvc

Below is the /logs folder ownership and permission. Please notice the 's' in drwxrwsrwx

drwxrwsrwx    3 root     root          4096 Oct  1 11:50 logs

Below is the content inside logs folder. Please notice the access folder is not created or deleted.

/logs # ls -lrt
total 4
drwxr-sr-x    2 root     root          4096 Oct  1 11:50 boot
/logs #

Below is the configmap of Fluent-Bit

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: eretail
  labels:
    k8s-app: fluent-bit
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush                     5
        Log_Level                 info
        Daemon                    off
        Parsers_File              parsers.conf
        HTTP_Server               On
        HTTP_Listen               0.0.0.0
        HTTP_Port                 2020
        
    @INCLUDE application-log.conf
  
  application-log.conf: |
    [INPUT]
        Name                tail
        Path                /logs/boot/*.log
        Tag                 boot
        
    [INPUT]
        Name                tail
        Path                /logs/access/*.log
        Tag                 access
        
    [OUTPUT]
        Name                cloudwatch_logs
        Match               *boot*
        region              ap-southeast-1
        log_group_name      eks-fluent-bit
        log_stream_prefix   boot-log-
        auto_create_group   On
        
    [OUTPUT]
        Name                cloudwatch_logs
        Match               *access*
        region              ap-southeast-1
        log_group_name      eks-fluent-bit
        log_stream_prefix   access-log-
        auto_create_group   On
        
  parsers.conf: |
    [PARSER]
        Name                docker
        Format              json
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%LZ

Below is error log of Fluent-bit container

AWS for Fluent Bit Container Image Version 2.14.0
Fluent Bit v1.7.4
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/10/01 06:20:33] [ info] [engine] started (pid=1)
[2021/10/01 06:20:33] [ info] [storage] version=1.1.1, initializing...
[2021/10/01 06:20:33] [ info] [storage] in-memory
[2021/10/01 06:20:33] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/10/01 06:20:33] [error] [input:tail:tail.1] read error, check permissions: /logs/access/*.log
[2021/10/01 06:20:33] [ warn] [input:tail:tail.1] error scanning path: /logs/access/*.log
[2021/10/01 06:20:38] [error] [net] connection #33 timeout after 5 seconds to: 169.254.169.254:80
[2021/10/01 06:20:38] [error] [net] socket #33 could not connect to 169.254.169.254:80

CodePudding user response：

There are multiple errors in your log. Anyway, pertain the directory issue, can you add

VOLUME $LOG_DIR/boot $LOG_DIR/access

AFTER the line EXPOSE 8177 in the Dockerfile, rebuild and re-run and see this resolve the issue?