Communicate to cluster that Spark History server is running-CodePudding

I have a working Spark cluster, with a master node and some worker nodes running on Kubernetes. This cluster has been used for multiple spark submit jobs and is operational.

On the master node, I have started up a Spark History server using the $SPARK_HOME/sbin/start-history-server.sh script and some configs to determine where the History Server's logs should be written:

spark.eventLog.enabled=true
spark.eventLog.dir=...
spark.history.fs.logDirectory=...
spark.hadoop.fs.s3a.access.key=...
spark.hadoop.fs.s3a.secret.key=...
spark.hadoop.fs.s3a.endpoint=...
spark.hadoop.fs.s3a.path.style.access=true

This was done a while after the cluster was operational. The server is writing the logs to an external DB (minIO using the s3a protocol).

Now, whenever I submit spark jobs it seems like nothing is being written away in the location I'm specifying.

I'm wondering about the following: How can the workers know I have started up the spark history server on the master node? Do I need to communicate this to the workers somehow?

Possible causes that I have checked:

No access/permissions to write to minIO: This shouldn't be the case as I'm running spark submit jobs that read/write files to the same minIO using the same settings
Logs folder does not exist: I was getting these errors before, but then I created a location for the files to be written away and since then I'm not getting issues
spark.eventLog.dir should be the same as spark.history.fs.logDirectory: they are

CodePudding user response：

Just found out the answer: the way your workers will know where to store the logs is by supplying the following configs to your spark-submit job:

spark.eventLog.enabled=true
spark.eventLog.dir=...
spark.history.fs.logDirectory=...

It is probably also enough to have these in your spark-defaults.conf on the driver program, which is why I couldn't find a lot of info on this as I didn't add it to my spark-defaults.conf.