I just learned the Spark, there is a most simple example, but I run in the Spark quote above errors,

 package example 

Import the typesafe. Scalalogging. LazyLogging 
The import org. Apache. Spark. {SparkConf, SparkContext} 

The object SparkExample extends LazyLogging {

Private val master="spark://master: 7077 
Private val appName="example - spark" 
Private val stopWords=Set (" GIF ", "PNG") 

Def main (args: Array [String]) : Unit={

Val conf=new SparkConf () 
. SetMaster (master) 
. SetAppName (appName) 

Val sc=new SparkContext (conf) 

Val lines=sc. TextFile (" file:/home/qiujl/access log ") 
Val wordsCount=wordsCount. Count (sc, lines, stopWords) 

Val counts=wordsCount. Collect (). MkString ("/", ", ", "") 
Logger. The info (counts) 
} 
} 

The following error:
[QWWSL @ master ~] $spark - submit - master spark://master: 48-7077 - num - executors driver - 2 g memory -- executor - 7 g - executor - memory cores 3/home/qiujl/example - spark - master. Jar
18/05/30 10:27:38 WARN NativeCodeLoader: Unable to load the native - hadoop library for your platform... Using the builtin - Java classes where applicable
Java. Lang. ClassNotFoundException: example. SparkExample
The at java.net.URLClassLoader.findClass URLClassLoader. Java: (381)
The at Java. Lang. This. LoadClass (424). This Java:
The at Java. Lang. This. LoadClass (357). This Java:
The at Java. Lang. Class. ForName0 (Native Method)
The at Java. Lang. Class.forname (348) Class. Java:
The at org. Apache. Spark. Util. Utils $. ClassForName (Utils. Scala: 235)
The at org.apache.spark.deploy.SparkSubmit$.org $$$$$$runMain SparkSubmit deploy spark apache (SparkSubmit. Scala: 836)
The at org. Apache. Spark. Deploy. SparkSubmit $. DoRunMain $1 (197). SparkSubmit scala:
The at org. Apache. Spark. Deploy. SparkSubmit $. Submit (SparkSubmit. Scala: 227)
The at org. Apache. Spark. Deploy. SparkSubmit $. The main (SparkSubmit. Scala: 136)
The at org. Apache. Spark. Deploy. SparkSubmit. Main (SparkSubmit. Scala)

CodePudding user response:

Java. Lang. ClassNotFoundException: example. SparkExample

Your code is not into the jar

CodePudding user response:

reference 1st floor tianfang response:

Java lang. ClassNotFoundException: example. SparkExample

Your code is not into the jar

can elaborate on it? I've just self-study, don't know much about it

CodePudding user response:

Packaging when added to the code, command specifies which class, -- class example. SparkExample

CodePudding user response:

reference wR__500 reply: 3/f

package when added to the code, command specifies which class, -- class example. SparkExample

Can you elaborate on it? Write in your code? Or configuration?

CodePudding user response:

Look at the situation is, when you pack, default to specify the main class, you can see how to pack the default specified methods, also can spark the default local executive, don't have to submit to the server, can use the val sparkConf=new sparkConf () setAppName (" SparkTest "). SetMaster (" local "), you can perform under the Windows development environment; You may have a look the contents of the package, you can find this class, can use the jd - GUI view jars.

CodePudding user response:

Submit the need to specify the jar package, at the same time put the jar package uploaded to the server of the spark,
Such as ${SPARK_HOME}/bin/spark - submit - master spark://10.70.64.15:8080 - jars./lib/*. The jar - class org. Apache. Sparkexamples. SparkSQL2./jar first. The jar