I've created a Flink cluster using Session mode on native K8s using the command:

$ ./bin/kubernetes-session.sh -Dkubernetes.cluster-id=my-first-flink-cluster

based on these instructions.

I am able to submit jobs to the cluster and view the Flink UI. However, I noticed that Flink creates a taskmanager pod only when the job is submitted and deletes it right after the job is finished. Previously I tried the same using YARN based deployment on Google Dataproc and with that method the cluster had a taskmanager always running which reduced job start time.

Hence, is there a way to keep a taskmanager pod always running using K8s Flink deployment such that job start time is reduced?

CodePudding user response：

the intention of the native k8s support provided by Flink is to have this active resource allocation (i.e. task slots through new TaskManager instances) in case it is needed. In addition to that, it will allow the shutdown of TaskManager pods if they are not used anymore. That's the behavior you're observing.

What you're looking for is the standalone k8s support. Here, Flink does not try to start new TaskManager pods. The ResourceManager is passive, i.e. it only considers the TaskManagers that are registered. Some outside process (or a user) has to manage TaskManager pods instead. This might lead to jobs failing if there are not enough task slots available.

Best, Matthias