Set (" spark. Executor. Instances ", "16")
Set (" spark. Executor. Cores ", "3")
Set (" spark. Executor. The memory ", "16 g")

Broadcast (Hashmap structure)
Hashmap data structure to store between 40-50 MB, how can appear OOM?

Arguably not each machine backup a copy of the data, each are 50 MB, 16 nodes

CodePudding user response:

Set (" spark. Executor. The memory ", "16 g") although this is set up 16 g, but also want to see you on the node memory really left, if the value of the remaining 20 MB, you set here is so big, also have a role

CodePudding user response:

Broadcast is a data in each executor, not on each node, on the other hand, are you sure OOM was caused by a broadcast?