Small white wrote a team number activity score model, there is no problem on the IDEA of running, run 100 number tens of seconds, but a into the cluster, the code has been running half an hour, then finally fail to run, check the log display is out of memory, do you have a great god met this problem? O solve, has been confused two days, before practicing mlib, there are two methods also has this kind of problem, the local can run, on a cluster of memory,
Online said 浭 water spark - defaults. The spark inside the conf. Driver. ExtraJavaOptions - XX: PermSize=128 m - XX: MaxPermSize=256 m, I changed to or wrong,
The log content is as follows:

CodePudding user response:

I'm sorry, because of personal ability is limited, can't help you,

CodePudding user response:

Cluster on each node how much is allocated memory space, generally the default is 1 g, also need to check the driver set the size of the memory

CodePudding user response:

Driver is generally 512 m - 1 g, executor formula can be calculated, is the core number commonly: memory is 1:2 or 1:4, a single Worker node can be allocated more Worker instance, generally is the core number/4, in the spark - env. Sh configuration inside
In addition you what is this complex model? Article 100 can run so long,,, consider optimizing code