pod is not changing state from unready to ready-CodePudding

I am using the readiness probe feature in my k8s pod. For this, probe checks if the file /tmp/healthy is present or not. If not, probe mark pod UN-READY. I have observed If I create the file again /tmp/healthy, the pod does not come back to READY state. Does it mean, in pod lifecycle there is only path of READY--> UN-READY but not vice-versa?

/home/ravi/for_others/ric/stub>kubectl describe pod -n myns  deployment-eterm
Name:         deployment-eterm
Namespace:    myns
Priority:     0
Status:       Running
IP:           192.168.252.87
IPs:
  IP:           192.168.252.87
Controlled By:  ReplicaSet/deployment-myns-eterm-micro-6b896556c8
Containers:
    Liveness:       exec [/bin/sh -c /tmp/liveliness-status-collector.sh] delay=60s timeout=1s period=2s #success=1 #failure=5
    Readiness:      exec [/bin/sh -c cat /tmp/healthy] delay=60s timeout=1s period=10s #success=1 #failure=1
    Environment Variables from:
      configmap-myns-eterm-env-micro  ConfigMap  Optional: false
    Environment:                       <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-57jpz (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age               From                    Message
  ----     ------     ----              ----                    -------
  Normal   Scheduled  9m11s             default-scheduler       Successfully assigned myns/deployment-eterm to mnode
  Normal   Pulled     9m10s             kubelet, mnode  Container image "porics.microlab.com:5000/eterm:latest" already present on machine
  Normal   Created    9m10s             kubelet, mnode  Created container container-myns-eterm
  Normal   Started    9m10s             kubelet, mnode  Started container container-myns-eterm
  Normal   Pulled     9m10s             kubelet, mnode  Container image "porics.microlab.com:5000/config-proxy:latest" already present on machine
  Normal   Created    9m10s             kubelet, mnode  Created container container-myns-configproxy
  Normal   Started    9m10s             kubelet, mnode  Started container container-myns-configproxy
  Warning  Unhealthy  3s (x3 over 23s)  kubelet, mnode  Readiness probe failed: cat: /tmp/healthy: No such file or directory
/home/ravi/for_others/ric/stub>

CodePudding user response：

The below statement is not true:

Does it mean, in pod lifecycle, there is the only one path of READY--> UN-READY but not vice-versa?

You can run the below experiment to see the pod transition:

Create a pod with below manifest file:

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: readiness
  name: readiness-test
spec:
  nodeName: kube-master
  containers:
  - name: liveness
    image: bash
    command: ['bash','-c', 'while true;do echo "$(date): creating /tmp/healthy"; touch /tmp/healthy; sleep 10; echo "$(date): deleting /tmp/healthy";rm /tmp/healthy ;sleep 10;done']
    readinessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 2

The above manifest will cause the pod to transition into ready/not-ready every 10 seconds.

kubectl get pod -w
NAME             READY   STATUS    RESTARTS   AGE
readiness-test   0/1     Running   0          79s
readiness-test   1/1     Running   0          83s
readiness-test   0/1     Running   0          97s
readiness-test   1/1     Running   0          103s
readiness-test   0/1     Running   0          117s
readiness-test   1/1     Running   0          2m3s
readiness-test   0/1     Running   0          2m17s
readiness-test   1/1     Running   0          2m23s
readiness-test   0/1     Running   0          2m37s
readiness-test   1/1     Running   0          2m43s
readiness-test   0/1     Running   0          2m57s
readiness-test   1/1     Running   0          3m3s
readiness-test   0/1     Running   0          3m17s
readiness-test   1/1     Running   0          3m23s
readiness-test   0/1     Running   0          3m37s
readiness-test   1/1     Running   0          3m43s
readiness-test   0/1     Running   0          3m57s
readiness-test   1/1     Running   0          4m3s

You can see the the readiness status getting changed:

while true; do k get pod readiness-test -o jsonpath='{.status.containerStatuses[*].ready}{"\n"}';sleep 2 ;done
true
true
true
true
true
true
false
false
true
true
true
true
true
true
false
false
true
true
true
true
true
true
false
false
true
true
true
true
true
false
false
false