My goal is to have a kubernetes cluster running with stateless replicas of some kind of frontend which can schedule jobs.

Afterwards my plan is to have multiple runners (as pods) which are polling for scheduled jobs. Once they receive job data they should launch job executors (also as a pod) on demand.

This should look somehow like this:

  pod A                    pod B
 ________                __________
|        |      kube    |          |
| runner |    ------>   | executor |  
|________|              |__________|
                 .
                 .
  pod A'         .         pod B'
 ________                __________
|        |      kube    |          |
| runner |    ------>   | executor |  
|________|              |__________|

Basically I am kind of inspired by the gitlab architecture.