I have a Spring Boot application running in a Docker container managed by Kubernetes whose job is to pull data from a queue, process it, and write back to a queue. Initially, the Deployment for this application is created with 1 replica. However, at some point I need to scale that Deployment up in order to process messages from the queue faster. Ideally, I'd like to have that existing 1 replica start processing messages, and then the new replicas can come up and help out.
The problem I've discovered is that when scaling the Deployment programmatically, the existing 1 replica is destroyed, and this typically is occurring in the middle of it processing some data, resulting in the output of that processing being lost.
Example of what I'd like to see happen:
- One replica processing message
- Deployment gets scaled to 5 replicas
- Four new replicas are spun up to help process messages leaving the existing.
Is this at all possible? Thanks!
CodePudding user response：
Normally, Kubernetes scales up without restarting the existing Pods, so kind of unusual that this does not happen in your case.
That aside though, none of your Pods should shut down in a way that results in loss of messages. This suggests that you are missing code for graceful shutdown.
With Spring Boot and K8s you will have to do two things:
- Introduce a shutdown handler in Spring Boot that is triggered by K8s initial SIGTERM signal and makes sure the application does not accept new messages, but gets time to handle in-flight messages. Spring Boot provides several mechanisms for this.
- Makes sure that K8s gives your application sufficient time for graceful shutdown by adequately configuring the terminationGracePeriodSeconds. This is the time your applications gets to shut itself down in an orderly manner before K8s sends the SIGKILL signal.
CodePudding user response：
As David Maze suggested, I was not only patching the number of replicas but also patching an environment variable in the template, causing the replica to be recreated in order to pick that new variable up.
Fritz's answer is also a great overall suggestion for a graceful shutdown.