How can I use Prometheus (Which uses a 'pull' approach) to monitor my Spark clusters when I spin up a new cluster every time I need, which means I don't have one steady monitoring endpoint.

CodePudding user response：

It depends a bit on how / where you spin up your clusters. When using Kubernetes, you can use Kubernetes SD configurations to discover your targets or use the Prometheus Operator (see also here).

If running on EMR you could use bootstrap actions to configure the node_exporter and jmx_exporter. See this for an example how to do so.

And alternatively, you could use the Prometheus Pushgateway together with a custom Spark sink (example) for this use case.