I am new to EMR and I am running an EMR cluster, with 1 master (32gb) and 5 core nodes (16gb). I launch 11 apps. The apps have to be separated in case one of them fail (all of them are streaming apps). I must mention that I also got ElasticSearch running on the cluster. After some time the master node is running out of memory and stops responding and some apps starting to fail. In the process overview I found many smaller hadoop processes with that occupy 1-1.3GB of RAM. I guess these are the driver processes from each app. I tried to reduce the the driver memory under "spark.driver.memory" to 512MB, but it's still at 1.3GB after relaunching the apps. Is this because of yarn? ES just allocates ca. 6.5 GB of RAM of the master node

CodePudding user response：

I had to specify the driver memory in spark-submit command like this: spark-submit --driver-memory 500M

because to specify it inside the python file is too late, when you run the driver in client mode, because it allocates the memory before