Home > other >  Spark streaming saveAsNewAPIHadoopDataset method is used to write in the hbase, recover from checkpo
Spark streaming saveAsNewAPIHadoopDataset method is used to write in the hbase, recover from checkpo

Time:09-27

Recently wrote a read data from the Kafka, after processing by writing to the hbase saveAsNewAPIHadoopDataset method, normal operation without error, write also is normal, but when the application is stopped manually, again (through the Checkpoint recovery) would be an error, the great god answer! Genuflect is begged! Error message is as follows:
15/12/22 16:26:52 WARN VerifiableProperties: Property serializer. The class is not valid
15/12/22 16:26:57 WARN FileOutputCommitter: the Output Path is null in setupJob ()
15/12/22 16:26:58 WARN TaskSetManager: Lost task in stage 0.0 1.0 (dar 1, 10.4.120.183) : Java. Lang. NullPointerException
At org, apache hadoop. Fs. Path. & lt; init> (105) Path. Java:
At org, apache hadoop. Fs. Path. & lt; init> (94) Path. Java:
At org, apache hadoop. Graphs. Lib. Output. FileOutputFormat. GetDefaultWorkFile (FileOutputFormat. Java: 286)
At org, apache hadoop. Graphs. Lib. Output. TextOutputFormat. GetRecordWriter (TextOutputFormat. Java: 129)
The at org. Apache. Spark. RDD. PairRDDFunctions $$$saveAsNewAPIHadoopDataset anonfun $1 $$anonfun $12. Apply (PairRDDFunctions. Scala: 1030)
The at org. Apache. Spark. RDD. PairRDDFunctions $$$saveAsNewAPIHadoopDataset anonfun $1 $$anonfun $12. Apply (PairRDDFunctions. Scala: 1014)
The at org. Apache. Spark. The scheduler. ResultTask. RunTask (ResultTask. Scala: 66)
At org. Apache. Spark. The scheduler. Task. Run (88) Task. Scala:
The at org. Apache. Spark. Executor. $TaskRunner executor. Run (executor. Scala: 214)
The at Java. Util. Concurrent. ThreadPoolExecutor. RunWorker (ThreadPoolExecutor. Java: 1145)
The at Java. Util. Concurrent. ThreadPoolExecutor $Worker. The run (ThreadPoolExecutor. Java: 615)
The at Java. Lang. Thread. The run (Thread. Java: 722)

^ C15/12/22 16:26:59 ERROR TaskSetManager: Task 0 in stage 1.0 failed 4 times; Aborting job
15/12/22 16:26:59 ERROR JobScheduler: ERROR running job streaming job 1450772680000 ms. 0
Org. Apache. Spark. SparkException: Job aborted due to stage a failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost Task in stage 0.3 1.0 (dar 4, 10.4.120.183) : Java. Lang. NullPointerException
At org, apache hadoop. Fs. Path. & lt; init> (105) Path. Java:
At org, apache hadoop. Fs. Path. & lt; init> (94) Path. Java:
At org, apache hadoop. Graphs. Lib. Output. FileOutputFormat. GetDefaultWorkFile (FileOutputFormat. Java: 286)
At org, apache hadoop. Graphs. Lib. Output. TextOutputFormat. GetRecordWriter (TextOutputFormat. Java: 129)
The at org. Apache. Spark. RDD. PairRDDFunctions $$$saveAsNewAPIHadoopDataset anonfun $1 $$anonfun $12. Apply (PairRDDFunctions. Scala: 1030)
The at org. Apache. Spark. RDD. PairRDDFunctions $$$saveAsNewAPIHadoopDataset anonfun $1 $$anonfun $12. Apply (PairRDDFunctions. Scala: 1014)
The at org. Apache. Spark. The scheduler. ResultTask. RunTask (ResultTask. Scala: 66)
At org. Apache. Spark. The scheduler. Task. Run (88) Task. Scala:
The at org. Apache. Spark. Executor. $TaskRunner executor. Run (executor. Scala: 214)
The at Java. Util. Concurrent. ThreadPoolExecutor. RunWorker (ThreadPoolExecutor. Java: 1145)
The at Java. Util. Concurrent. ThreadPoolExecutor $Worker. The run (ThreadPoolExecutor. Java: 615)
The at Java. Lang. Thread. The run (Thread. Java: 722)

Driver stacktrace:
The at org.apache.spark.scheduler.DAGScheduler.org $$$$$$failJobAndIndependentStages DAGScheduler scheduler spark apache (DAGScheduler. Scala: 1280)
The at org. Apache. Spark. The scheduler. DAGScheduler $$$abortStage anonfun $1. Apply (DAGScheduler. Scala: 1268)
The at org. Apache. Spark. The scheduler. DAGScheduler $$$abortStage anonfun $1. Apply (DAGScheduler. Scala: 1267)
At the scala. Collection. The mutable. ResizableArray $class. Foreach (ResizableArray. Scala: 59)
At the scala. Collection. Mutable. ArrayBuffer. Foreach (ArrayBuffer. Scala: 47)
The at org. Apache. Spark. The scheduler. DAGScheduler. AbortStage (DAGScheduler. Scala: 1267)
The at org. Apache. Spark. The scheduler. DAGScheduler $$$handleTaskSetFailed anonfun $1. Apply (DAGScheduler. Scala: 697)
The at org. Apache. Spark. The scheduler. DAGScheduler $$$handleTaskSetFailed anonfun $1. Apply (DAGScheduler. Scala: 697)
At the scala. Option. Foreach (236) Option. The scala:
The at org. Apache. Spark. The scheduler. DAGScheduler. HandleTaskSetFailed (DAGScheduler. Scala: 697)
The at org. Apache. Spark. The scheduler. DAGSchedulerEventProcessLoop. DoOnReceive (DAGScheduler. Scala: 1493)
The at org. Apache. Spark. The scheduler. DAGSchedulerEventProcessLoop. OnReceive (DAGScheduler. Scala: 1455)
The at org. Apache. Spark. The scheduler. DAGSchedulerEventProcessLoop. OnReceive (DAGScheduler. Scala: 1444)
The at org. Apache. Spark. Util. EventLoop $$$1. -anon run (EventLoop. Scala: 48)
The at org. Apache. Spark. The scheduler. DAGScheduler. RunJob (DAGScheduler. Scala: 567)
The at org. Apache. Spark. SparkContext. RunJob (SparkContext. Scala: 1813)
The at org. Apache. Spark. SparkContext. RunJob (SparkContext. Scala: 1826)
The at org. Apache. Spark. SparkContext. RunJob (SparkContext. Scala: 1903)
The at org. Apache. Spark. RDD. PairRDDFunctions $$$saveAsNewAPIHadoopDataset anonfun $1. Apply $MCV $sp (PairRDDFunctions. Scala: 1055)
The at org. Apache. Spark. RDD. PairRDDFunctions $$$saveAsNewAPIHadoopDataset anonfun $1. Apply (PairRDDFunctions. Scala: 998)
The at org. Apache. Spark. RDD. PairRDDFunctions $$$saveAsNewAPIHadoopDataset anonfun $1. Apply (PairRDDFunctions. Scala: 998)
The at org. Apache. Spark. RDD. RDDOperationScope $. WithScope (RDDOperationScope. Scala: 147)
nullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnull
  • Related