I am simply trying to call a method from a class (in a databricks notebook) I have created in Scala. Don't worry about all the code in between. Just focus on the SecondExplode
class and the registerUDF
function at the very bottom. The class is as follows:
case class SecondExplodeTimes(start_dt: String, end_dt: String, counter: Int)
class SecondExplode extends Serializable {
def expandDatetimeRangeToStartAndEndSeconds(start: String, end: String, ceilingLimit: Int): Seq[SecondExplodeTimes] =
{
val start_time: DateTime = DateTime.parse(start, DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss"))
val end_time: DateTime = DateTime.parse(end, DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss"))
val secBetween = end_time.getSecondOfMinute - start_time.getSecondOfMinute
(1 to secBetween).zipWithIndex.map{case (p, counter) => {
val strt = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss").print(start_time.plusSeconds(p-1))
val endt = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss").print(start_time.plusSeconds(p))
SecondExplodeTimes(strt, endt, counter)
}}.toSeq
}
def registerUDF: UserDefinedFunction = {
val spark = SparkSession.builder().getOrCreate()
spark.udf.register("second_explode", expandDatetimeRangeToStartAndEndSeconds _)
}
}
calling the class and function in the next cell
val secondexplode = new SecondExplode.registerUDF
I am getting this error:
error: not found: value SecondExplode. val secondexplode = new SecondExplode.registerUDF
I believe I am using version Scala 2 so the syntax is correct. Simply put, I am attempting to call the registerUDF
function from the SecondExplode
class to test it actually it works. Thank you in advance.
CodePudding user response:
I believe what you want is (new SecondExplode).registerUDF
.
The problem is with parsing precedence. The .
has higher precedence than new
. The compiler assumes you're trying to make a new
instance of SecondExplode.registerUDF
, rather than make a new SecondExplode
and then call the registerUDF
method on the instance.
The reason why this is a possible misinterpretation is that there's nothing that prevents you from doing:
object SecondExplode {
class registerUDF {}
}
in which case, new SecondExplode.registerUDF
would have worked.