Sparksubmit submit tasks to yarn, how to get to the task by means of code from the accept state transitions to the running state of time node and the node from the running status to the finish status, through sc before get appid, then through yarnclient to get the task status, but have not been able to listen to the accept state, is a great god, and of all stripes who have encountered similar problems

CodePudding user response:

Go to spark UI

CodePudding user response:

reference 1st floor DanielMaster response:

go to the UI spark

I want to through the way of code implementation

CodePudding user response:

Code to access the UI interface and analytical data

CodePudding user response:

reference ainibc reply: 3/f

code to access the UI interface and then parse the data

Code access will be a delay, is it possible to through the yarn internal implementation

CodePudding user response:

Access to the yarn restful interface
https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

CodePudding user response:

OnApplicationStart from https://spark.apache.org/docs/latest/api/java/index.html? Org/apache/spark/scheduler/SparkListener HTML