Amazon Deequ (Spark Scala ) - java.lang.NoSuchMethodError: 'scala.Option org.apache.spark.sql-CodePudding

Spark Version - 3.0.1 Amazon Deequ version - deequ-2.0.0-spark-3.1.jar

Im running the below code in spark shell in my local :

import com.amazon.deequ.analyzers.runners.{AnalysisRunner, AnalyzerContext}   
import com.amazon.deequ.analyzers.runners.AnalyzerContext.successMetricsAsDataFrame  
import com.amazon.deequ.analyzers.{Compliance, Correlation, Size, Completeness, Mean, 
ApproxCountDistinct, Maximum, Minimum, Entropy}  

import com.amazon.deequ.analyzers.{Compliance, Correlation, Size, Completeness, Mean, 
ApproxCountDistinct, Maximum, Minimum, Entropy}

val analysisResult: AnalyzerContext = {AnalysisRunner.onData(datasourcedf).addAnalyzer(Size()).addAnalyzer(Completeness("customerNumber")).addAnalyzer(ApproxCountDistinct("customerNumber")).addAnalyzer(Minimum("creditLimit")).addAnalyzer(Mean("creditLimit")).addAnalyzer(Maximum("creditLimit")).addAnalyzer(Entropy("creditLimit")).**run()**}

ERROR:

    java.lang.NoSuchMethodError: 'scala.Option 
org.apache.spark.sql.catalyst.expressions.aggregate.AggregateFunction.toAggregateExpression$default$2()'
at org.apache.spark.sql.DeequFunctions$.withAggregateFunction(DeequFunctions.scala:31)
at org.apache.spark.sql.DeequFunctions$.stateful_approx_count_distinct(DeequFunctions.scala:60)
at com.amazon.deequ.analyzers.ApproxCountDistinct.aggregationFunctions(ApproxCountDistinct.scala:52)
at com.amazon.deequ.analyzers.runners.AnalysisRunner$.$anonfun$runScanningAnalyzers$3(AnalysisRunner.scala:319)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
at scala.collection.immutable.List.flatMap(List.scala:355)
at com.amazon.deequ.analyzers.runners.AnalysisRunner$.liftedTree1$1(AnalysisRunner.scala:319)
at com.amazon.deequ.analyzers.runners.AnalysisRunner$.runScanningAnalyzers(AnalysisRunner.scala:318)
at com.amazon.deequ.analyzers.runners.AnalysisRunner$.doAnalysisRun(AnalysisRunner.scala:167)
at com.amazon.deequ.analyzers.runners.AnalysisRunBuilder.run(AnalysisRunBuilder.scala:110)
... 63 elided

Can someone please let me know how to resolve this issue

CodePudding user response：

You can't use Deeque version 2.0.0 with Spark 3.0 because it's binary incompatible due of the changes in the Spark's internals. With Spark 3.0 you need to use version 1.2.2-spark-3.0