Am using maven project with spark, and a cassandra as a database. Am able to execute the code with 0 error, and getting the result that I want, but I can't submit the project with the jar file: here is my pom.xml :
my commands to submit the project:
spark-submit --master "local[*]" --class com.sparkfinal.App target/sparkfinal-1.0-SNAPSHOT.jar
the error:
CodePudding user response：
You need either:
- create an assembly jar for your project that will include all dependencies (make sure you declare Spark dependencies as
provided) - or specify Spark Cassandra Connector as dependency when submitting the job by using
--packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0as shown in the connector's docs
I would recommend to use first method because I see that you have other dependencies that you'll need to provide. Also, please note that you may not need to specify java driver dependency as it will be pulled by Spark Cassandra Connector automatically.