I have changed the terminationGracePeriodSeconds from 30 to 120 in Kubernetes deployment manifest file but when I am deploying using helm (helm upgrade --install --values , the old pods getting terminated immediately and new pods started running.

But the expected behaviour is to keep the old pods in terminating state and continue its current processes for 120 seconds as defined,

What else could be missing here??

does this solve my issue here?

containers:
  - name: containername
    lifecycle:
      preStop:
        exec:
          command: [ "/bin/sleep", "20" ]

One question I had is does adding sleep command stops execution of current processes of the pod and just sleeps while it is in terminating state???

CodePudding user response：

The Kubernetes parameter terminationGracePeriodSeconds decides on how much time Kubernetes waits until it forcefully kills your container. In other words, when Kubernetes wants to terminate your Pod, it does the following:

1. Kubernetes sends SIGTERM to your container
2. Kubernetes waits max. terminationGracePeriodSeconds for the following sequence of operations to happen:
   • Container stops normally
   • PreStop hook executes
3. Kubernetes sends SIGKILL to your container (and your PreStop container) and therefore terminates them forcefully

Note that your container and the PreStop containers are not executed at the same time, but synchronously, one by one.

Read more at Kubernetes Hook Handler Execution.

CodePudding user response：

Basically, it is expected behavior for terminationGracePeriodSeconds, as it is optional duration in seconds the pod needs to terminate gracefully.

On pod termination - the lifecycle is described here - pod receives terminating state and it is removed from the endpoints list of all services and stops getting new traffic. SIGTERM signal is immediately sent to the processes running in pod, and Kubernetes is waiting for the pod to stop normally on its own during the grace period specified in terminationGracePeriodSeconds option. As soon as terminationGracePeriodSeconds expires, the Pod is forcefully killed by SIGKILL signal. In your case, the processes in the pod were just shutdown normally before 120 seconds of grace period have passed.

In its turn, preStop hook is called immediately before a pod is terminated, which means that the hook will be executed prior to kubectl sends SIGTERM to the pod. As stated in documentation, the hook must complete before the TERM signal to stop the container is sent. terminationGracePeriodSeconds is happening in parallel with the preStop hook and the SIGTERM signal and its countdown begins before the preStop hook is executed. The pod will eventually terminate within the pod's termination grace period, regardless of the hook completion.

Therefore, on receiving sleep command for preStop hook, pod already marked as Terminating but the processes inside of it are not yet terminated - so the container can complete all the active requests during this period.