Home > other >  A spark of ClassNotFound problem
A spark of ClassNotFound problem

Time:09-25

The source code is as follows:
Package org. Apache. Spark. Examples

The import scala. Math. The random

The import org. Apache. Spark. _

/* * Computes an approximation to PI */
The object SparkPi {
Def main (args: Array [String]) {
Val conf=new SparkConf (.) setAppName (" Spark Pi "). SetMaster (" Spark://192.168.70.24:7077 "). The set (" Spark. Executor. The memory ", "2 g")
Val spark=new SparkContext (conf)
Spark. AddJar (" D: \ \ workspace \ \ JAR \ \ SparkPi_jar \ \ SparkPi JAR ")
Val slices=the if (args. Length & gt; 0) the args (0). ToInt else 2
Val n=math. Min (100000 l * slices, Int. J MaxValue). ToInt//get overflow
Val count=spark. Parallelize (1 until n, slices). The map {I=& gt;
Val x=the random * 2-1
Val y=the random * 2-1
If (x * x + y * y & lt; 1) 1 else 0
}. Reduce + _) (_
Println (" Pi is roughly "+ 4.0 * count/n)
Spark. Stop ()
}
}

An error is as follows:
The Exception in the thread "main" org. Apache. Spark. SparkException: Job aborted due to stage a failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost Task in stage 0.0 0.3 (6, dar HadoopMaster) : Java. Lang. ClassNotFoundException: org. Apache. Spark. Examples. SparkPi $$anonfun $1
The at java.net.URLClassLoader.findClass URLClassLoader. Java: (381)
The at Java. Lang. This. LoadClass (424). This Java:
The at Java. Lang. This. LoadClass (357). This Java:
The at Java. Lang. Class. ForName0 (Native Method)
The at Java. Lang. Class.forname (348) Class. Java:
The at org. Apache. Spark. Serializer. JavaDeserializationStream $$$1. -anon resolveClass (JavaSerializer. Scala: 68)
The at Java. IO. ObjectInputStream. ReadNonProxyDesc (ObjectInputStream. Java: 1620)
The at Java. IO. ObjectInputStream. ReadClassDesc (ObjectInputStream. Java: 1521)
The at Java. IO. ObjectInputStream. ReadOrdinaryObject (ObjectInputStream. Java: 1781)
The at Java. IO. ObjectInputStream. ReadObject0 (ObjectInputStream. Java: 1353)
The at Java. IO. ObjectInputStream. DefaultReadFields (ObjectInputStream. Java: 2018)
The at Java. IO. ObjectInputStream. ReadSerialData (ObjectInputStream. Java: 1942)
The at Java. IO. ObjectInputStream. ReadOrdinaryObject (ObjectInputStream. Java: 1808)
The at Java. IO. ObjectInputStream. ReadObject0 (ObjectInputStream. Java: 1353)
The at Java. IO. ObjectInputStream. DefaultReadFields (ObjectInputStream. Java: 2018)
The at Java. IO. ObjectInputStream. ReadSerialData (ObjectInputStream. Java: 1942)
The at Java. IO. ObjectInputStream. ReadOrdinaryObject (ObjectInputStream. Java: 1808)
The at Java. IO. ObjectInputStream. ReadObject0 (ObjectInputStream. Java: 1353)
The at Java. IO. ObjectInputStream. DefaultReadFields (ObjectInputStream. Java: 2018)
The at Java. IO. ObjectInputStream. ReadSerialData (ObjectInputStream. Java: 1942)
The at Java. IO. ObjectInputStream. ReadOrdinaryObject (ObjectInputStream. Java: 1808)
The at Java. IO. ObjectInputStream. ReadObject0 (ObjectInputStream. Java: 1353)
The at Java. IO. ObjectInputStream. ReadObject (ObjectInputStream. Java: 373)
The at org. Apache. Spark. Serializer. JavaDeserializationStream. ReadObject (JavaSerializer. Scala: 76)
The at org. Apache. Spark. Serializer. JavaSerializerInstance. Deserialize (JavaSerializer. Scala: 115)
The at org. Apache. Spark. The scheduler. ResultTask. RunTask (ResultTask. Scala: 61)
At org. Apache. Spark. The scheduler. Task. Run (89) Task. Scala:
The at org. Apache. Spark. Executor. $TaskRunner executor. Run (executor. Scala: 214)
The at Java. Util. Concurrent. ThreadPoolExecutor. RunWorker (ThreadPoolExecutor. Java: 1142)
The at Java. Util. Concurrent. ThreadPoolExecutor $Worker. The run (ThreadPoolExecutor. Java: 617)
The at Java. Lang. Thread. The run (Thread. Java: 745)

Driver stacktrace:
The at org.apache.spark.scheduler.DAGScheduler.org $$$$$$failJobAndIndependentStages DAGScheduler scheduler spark apache (DAGScheduler. Scala: 1431)
The at org. Apache. Spark. The scheduler. DAGScheduler $$$abortStage anonfun $1. Apply (DAGScheduler. Scala: 1419)
The at org. Apache. Spark. The scheduler. DAGScheduler $$$abortStage anonfun $1. Apply (DAGScheduler. Scala: 1418)
At the scala. Collection. The mutable. ResizableArray $class. Foreach (ResizableArray. Scala: 59)
At the scala. Collection. Mutable. ArrayBuffer. Foreach (ArrayBuffer. Scala: 47)
The at org. Apache. Spark. The scheduler. DAGScheduler. AbortStage (DAGScheduler. Scala: 1418)
The at org. Apache. Spark. The scheduler. DAGScheduler $$$handleTaskSetFailed anonfun $1. Apply (DAGScheduler. Scala: 799)
The at org. Apache. Spark. The scheduler. DAGScheduler $$$handleTaskSetFailed anonfun $1. Apply (DAGScheduler. Scala: 799)
At the scala. Option. Foreach (236) Option. The scala:
The at org. Apache. Spark. The scheduler. DAGScheduler. HandleTaskSetFailed (DAGScheduler. Scala: 799)
The at org. Apache. Spark. The scheduler. DAGSchedulerEventProcessLoop. DoOnReceive (DAGScheduler. Scala: 1640)
The at org. Apache. Spark. The scheduler. DAGSchedulerEventProcessLoop. OnReceive (DAGScheduler. Scala: 1599)
The at org. Apache. Spark. The scheduler. DAGSchedulerEventProcessLoop. OnReceive (DAGScheduler. Scala: 1588)
The at org. Apache. Spark. Util. EventLoop $$$1. -anon run (EventLoop. Scala: 48)
The at org. Apache. Spark. The scheduler. DAGScheduler. RunJob (DAGScheduler. Scala: 620)
nullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnull
  • Related