What is the shufflequerystage box that I see in the spark DAGs. How is it different from the excahnge box in the spark stages?

CodePudding user response：

There is already a nice answer here, but this is just to give you some more info on what this shufflequerystage actually is by looking at the source code.

What is a Shuffle Query Stage?

If we look at Spark's source code for the

Then there is job4 which just reuse what was computed previously but adds additional 4th stage with ShuffleQueryStage similar as in your case