I have multiple libraries dependency in the sbt.build file. I am creating the final jar file using sbt-assembly so it includes all the dependent libraries in the jar itself.

But using jar tvf jarname.jar, I am not able to find all libraries there.

I need this to bundle all libraries in a jar and provide it to spark-shell with spark-shell --jar jarpath and then using the import command to use the libraries there.

This is done because this is not possible for me to import packages directly to spark-shell using the spark-shell --packages command.

Expected: Adding the jar file to the spark-shell and then importing all libraries there which should be present in jar

CodePudding user response：

Found the solution here:

Some of my dependencies includes the provided tag and thus it was not getting included in the fat jar.

libraryDependencies  = "org.apache.flink" %% "flink-table-planner" % flinkVersion % "provided"

CodePudding user response：

Look at The sbt-assembly Plugin

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.15.0")

sbt assembly

will give you a fat jar