Home > Software design >  Spark / scala : publish a single fat jar with sbt
Spark / scala : publish a single fat jar with sbt

Time:08-17

I am trying to deploy a single Fat Jar from a spark/scala project to a private nexus repository. The publish works fine but I get a large amount of files instead of one single assembly Jar :

  • *-assembly.jar
  • *-assembly.jar.md5
  • *-assembly.jar.sha1
  • *-javadoc.jar
  • *-javadoc.jar.md5
  • *-javadoc.jar.sha1
  • *-source.jar
  • *-source.jar.md5
  • *-source.jar.sha1
  • *.jar
  • *.jar.md5
  • *.jar.sha1
  • *-pom.jar
  • *-pom.jar.md5
  • *-pom.jar.sha1

My built.sbt is :

name         := "SampleApp"
version      := "0.1-SNAPSHOT"
scalaVersion := "2.12.14"

ThisBuild / useCoursier := false

libraryDependencies   = Seq(
  "com.github.scopt" %% "scopt"     % "4.0.1",
  "org.apache.spark" %% "spark-sql" % "3.1.1" % "provided"
)

resolvers   = Seq(
  "confluent" at "https://packages.confluent.io/maven/"
)

assembly / assemblyMergeStrategy := {
  case PathList("META-INF","services",xs @ _*) => MergeStrategy.filterDistinctLines
  case PathList("META-INF",xs @ _*) => MergeStrategy.discard
  case "application.conf" => MergeStrategy.concat
  case _ => MergeStrategy.first
}

assembly / assemblyExcludedJars := {
  val cp = (assembly / fullClasspath).value
  cp filter { f =>
    f.data.getName.contains("hadoop-hdfs-2")
    f.data.getName.contains("hadoop-client")
  }
}

assembly / artifact := {
  val art = (assembly / artifact).value
  art.withClassifier(Some("assembly"))
}

assembly / assemblyJarName := s"${name.value}-${version.value}.jar"

addArtifact(assembly / artifact, assembly)

resolvers  = ("Sonatype Nexus Repository Manager" at "http://localhost:8081/repository/app/").withAllowInsecureProtocol(true)
credentials  = Credentials("Sonatype Nexus Repository Manager", "localhost:8081", "user", "pass")

publishTo := {
  val nexus = "http://localhost:8081/repository/app/"
  if (isSnapshot.value) {
    Some("snapshots" at nexus   "test-SNAPSHOT")
 } else
    Some("releases"  at nexus   "test-RELEASE")
}

Is there a way to filter files with sbt before the publish in order to get only the *-assembly.jar ? Thanks a lot.

CodePudding user response:

This should disable the publish of unnecessary files in my case :

publishArtifact in (Compile, packageBin) := false
publishArtifact in (Compile, packageDoc) := false
publishArtifact in (Compile, packageSrc) := false
  • Related