Spark streaming docking kafka, then customize the output, save the HDFS, according to the documents content, an hour to a batch file, implementation file additional, using gzip compression output file, but after running, found the executor too few hours by yarn kill off, and then start a new, trace the line because the compression caused by direct memory memory leaks, because compression using Java NIO, using direct memory, seeks the solution!!!!!