Have more than one machine, only have 1 gb of memory, I good the spark environment, normal spark startup, check my process using the JPS master and the worker are launched, but running examples of spark program will be a problem, and submitted to the Task serialization failure 'Task serialization failed, whether the memory is too small, a great god for help

CodePudding user response:

First spark is a memory computing framework, internal have higher requirements, but 1 g single node installation is certainly no problem, run some small amount of calculation data and process is certainly can, start serialization failure may be the memory is too small, you can will be dependent on a part of the class cannot be serialized using transient keyword modifiers, tells the compiler to it does not require serialization,