I'm using AWS EKS Fargate to deploy my work. After applying the deployment yaml file, everything goes well in first 10mins, but after that, I failed to access the pod by using kubectl exec <podname> -- bash, when typing kubectl describe pod <podname>, both readinessProbe and livenessProbe return similar messages as below:

Events:
  Type     Reason           Age                 From               Message
  ----     ------           ----                ----               -------
  Warning  LoggingDisabled  16m                 fargate-scheduler  Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found
  Normal   Scheduled        15m                 fargate-scheduler  Successfully assigned k8s-fargate/k8s-api-5765846f76-d7nws to fargate-ip-10-0-130-250.ap-east-1.compute.internal
  Normal   Pulling          15m                 kubelet            Pulling image "awsaccid.dkr.ecr.ap-east-1.amazonaws.com/k8s-api-test:1.0.0"
  Normal   Pulled           14m                 kubelet            Successfully pulled image "awsaccid.dkr.ecr.ap-east-1.amazonaws.com/k8s-api-test:1.0.0" in 1m18.703187993s
  Normal   Created          14m                 kubelet            Created container k8s-api
  Normal   Started          14m                 kubelet            Started container k8s-api
  Warning  Unhealthy        2m18s               kubelet            Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "c2a2e9750a44684104a7e76a92bf7abe814ba29f306b092a48e17b90aab7f2dd": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: resource temporarily unavailable: unknown
  Warning  Unhealthy        2m13s               kubelet            Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "bcb60f638e2c364adc8694bc12f00660e2b0d7647d3861d3462727976d2df08c": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: resource temporarily unavailable: unknown
  Warning  Unhealthy        2m8s                kubelet            Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "988d29870b88fdcaae3cedf1071e79d2a786638c801364d71b6c7886f0be79e1": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: resource temporarily unavailable: unknown

Moreover, livenessProbe hasn't restart the pod even it is unhealthy.

I spent a whole day for this but still failed to solve it, anyone knows the problem? Thank you so much

Here's my deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: k8s-fargate
  name: k8s-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: k8s-api
  template:
    metadata:
      labels:
        app: k8s-api
    spec:
      volumes:
        - name: k8s-properties
          configMap: 
            name: k8s-properties
      containers:
        - name: k8s-api
          image: awsaccountid.dkr.ecr.ap-east-1.amazonaws.com/k8s-test:1.0.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8443
          resources:
            requests:
              memory: "1024Mi"
              cpu: "200m"
            limits:
              memory: "2500Mi"
              cpu: "1000m"
          volumeMounts:
            - name: k8s-properties
              mountPath: "/usr/local/folder"
              readOnly: false
          livenessProbe:
            exec:
              command:
              - cat
              - /usr/local/folder/file
            initialDelaySeconds: 5
            periodSeconds: 30
          readinessProbe:
            exec:
              command:
              - cat
              - /usr/local/folder/file
            initialDelaySeconds: 5
            periodSeconds: 5     

CodePudding user response：

Problem solved by creating new Docker image. Still have no idea of the error, but problem likely comes from the image container itself.