Kubernetes pod created after big delay by Job-CodePudding

I have a cron job that runs a job at 2:00 AM IST daily. Usually, the job creates the pod immediately but today the pod was created at 11:00 PM rather than 2:00 AM.

What can be the reasons for this big delay? How can I avoid that in the future?

CodePudding user response：

If the pod from a cronjob takes ages to appear, it can be a sign of a bad or orphaned webhook configuration. This question sounds very similar to another one I answered earlier this year: Kubernetes jobs are created but not executed immediately

Please check that one and see if matches your situation.

CodePudding user response：

The CronJob is only responsible for creating Jobs that match its schedule, and the Job in turn is responsible for the management of the Pods it represents.

Given the below, the delay might be either

Startup Performance

By defining start delay to be the wall time from expected cron start to application code actually executing. That is, if a cron is expected to run at 00:00:00, and the application code actually begins execution at 00:00:22, then the particular cron invocation has a start delay of 22 seconds. On the other hand, Kubernetes CronJobs can experience significant start delays because they require several events to happen prior to any application code beginning to run.

Cron job controller Processing Latency

The cron job controller implementation does so synchronously, issuing at least 1 additional API call for every CronJob. When the number of CronJobs exceeds a certain amount, these API calls begin to be rate-limited client-side. The latencies from the 10 second polling cycle and API client rate-limiting add up and contribute to a noticeable start-delay for CronJobs.

Scheduling Cron Pods If you do not provision compute for peak demand (and/or use a cloud-provider for compute instances) and instead use something like cluster autoscaler to dynamically scale nodes, then node launch times can contribute additional delays to launching CronJob Pods

Refer this Delay between k8s job created and k8s pod pending for more information