kubernetes init container for spark-submit-CodePudding

I am trying to run a spark-submit to the Kubernetes cluster with spark 3.2.1 image and it is working. Now my question is, can I execute an init container along with the spark-submit? What am trying to achieve is that the init container check another service is up or not, is it up then spark-submit will run or it fail.

I can see that a conf parameter "spark.kubernetes.initContainer.image" for spark version 2.3 but not for 3.2.1 (https://spark.apache.org/docs/2.3.0/running-on-kubernetes.html)

is there any mechanism that I can use to check other services are up or not before I submit a spark job?

I can see init container usage for the spark in the below links but it is not providing an accurate answer

https://docs.bitnami.com/kubernetes/infrastructure/spark/configuration/configure-sidecar-init-containers/ https://doc.lucidworks.com/spark-guide/11153/running-spark-on-kubernetes

any help will be much appreciated, thanks.

CodePudding user response：

I found that the best way to submit a spark job is the sparkoperator, more details can be found in the GitHub link

There is one option to include an init container and a sidecar container.

CodePudding user response：

You don't mention if the other service is in the same container or not but the principles are the same. It's covered in the docs here and gives this example which defines a simple Pod that has two init containers. The first waits for myservice, and the second waits for mydb. Once both init containers complete, the Pod runs the app container from its spec section.

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  containers:
  - name: myapp-container
    image: busybox:1.28
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
  initContainers:
  - name: init-myservice
    image: busybox:1.28
    command: ['sh', '-c', "until nslookup myservice.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for myservice; sleep 2; done"]
  - name: init-mydb
    image: busybox:1.28
    command: ['sh', '-c', "until nslookup mydb.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mydb; sleep 2; done"]