Recently there is a task to be continuously polling data into HDFS redis, calls to spark is simply a spark SQL insert table XXXX
Spark version 2.3.2, driver (16 g), 7 executor (8 g)
But rising is to discover this simple task executor of memory (driver not, RSS has to 5 g), but the JVM is no problem, the following is through vjtools vjtop monitoring screenshots, preliminary doubt is caused by Netty heap memory, but I don't know how to set in order to avoid this problem?


Run a day, rising memory, swap also began to use, such as swap out estimate executor will dead

CodePudding user response:

I also met, not solve, can only write a script to restart in the middle of the night once a day, and solutions, such as

CodePudding user response:

(1) continuous polling is an infinite loop?? Or external scheduling spark program?
Read quickly, and write slowly, or file and more small, lead to congestion, accumulation, memory continues to rise, to observe to observe the stages of the spark.