Spark Build Fails Because Of Avro Mapred Dependency-CodePudding

I have a scala spark project that fails because of some dependency hell. Here is my build.sbt:

scalaVersion := "2.13.3"

val SPARK_VERSION = "3.2.0"

libraryDependencies   = Seq(
  "com.typesafe" % "config" % "1.3.1",
  "com.github.pathikrit" %% "better-files" % "3.9.1",
  "org.apache.commons" % "commons-compress" % "1.14",
  "commons-io" % "commons-io" % "2.6",
  "com.typesafe.scala-logging" %% "scala-logging" % "3.9.4",
  "ch.qos.logback" % "logback-classic" % "1.2.3" exclude ("org.slf4j", "*"),
  "org.plotly-scala" %% "plotly-render" % "0.8.1",
  "org.apache.spark" %% "spark-sql" % SPARK_VERSION,
  "org.apache.spark" %% "spark-mllib" % SPARK_VERSION,

  // Test dependencies
  "org.scalatest" %% "scalatest" % "3.2.10" % Test,
  "com.amazon.deequ" % "deequ" % "2.0.0-spark-3.1" % Test,
  "org.awaitility" % "awaitility" % "3.0.0" % Test,
  "org.apache.spark" %% "spark-core" % SPARK_VERSION % Test,
  "org.apache.spark" %% "spark-sql" % SPARK_VERSION % Test

Here is the build failure:

[error] stack trace is suppressed; run 'last update' for the full output
[error] stack trace is suppressed; run 'last ssExtractDependencies' for the full output
[error] (update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar: not found: https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar
[error] (ssExtractDependencies) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar: not found: https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar
[error] Total time: 5 s, completed Dec 19, 2021, 5:14:33 PM
[info] shutting down sbt server

Is this caused by the fact that I',m using Scala 2.13?

CodePudding user response：

I had to do the inevitable and add this to my build.sbt:

ThisBuild / useCoursier := false