Spark a middle phase takes seriously-CodePudding

Among local run a program, the second phase should be seconds out of the results more than two minutes to the results, I don't know where there is an error, a great god can help correct?

CodePudding user response:

You point into the Stage, the Task of information just know

CodePudding user response:

reference 1st floor link0007 response:

into the Stage, you want to have a look at the Task information just know

to specific watching what data,,,, this is the point in the Task of information

CodePudding user response:

I wipe the sweat,,, you should be not understand the execution logic of the Spark, and the filter map flatMap operations such as belonging to the transform, RDD after several of the transform, until the action (e.g., take count isEmpty foreach foreachPartition) will actually perform, it doesn't make any sense to you calculate time 1

CodePudding user response:

reference link0007 reply: 3/f

I wipe the sweat,,, you should be not understand the execution logic of the Spark, and the filter map flatMap operations such as belonging to the transform, RDD after several of the transform, until the action (e.g., take count isEmpty foreach foreachPartition) will actually perform, it doesn't make any sense to you calculate time 1

Well about it but I also know that at the beginning just such a set up, in fact, the most at the start of the iteration three times running time is 30 seconds, but now I don't know why in every round of the second phase of time can achieve two minutes, iterative three times down the time for about ten minutes, I am ever before reduceByKey repatition among (3), then changed repartion (5), and later changed to repartion (200), the repartion should be to prevent the data skew I use it a node is single, didn't understand at the time, so add to the, after 200 found that takes too long, then deleted, but from then on, the task becomes 200, also has always been two minutes into the second phase time from the new set to repartition is (3), don't know how to return a responsibility