Convert timestamp to day-of-week string with date_format in Spark/Scala-CodePudding

I keep on getting error, that I pass too many arguments and not sure why, as I am following the exact examples from:

command-629173529675356:9: error: too many arguments for method apply: (colName: String)org.apache.spark.sql.Column in class Dataset
val df_2 = date_format.withColumn("day_of_week", date_format(col("date"), "EEEE"))

My code:

val date_format = df_filter.withColumn("date", to_date(col("pickup_datetime")))
val df_2 = date_format.withColumn("day_of_week", date_format(col("date"), "EEEE"))

Thank you for the help!

CodePudding user response：

dayofweek is the function you're looking for, so something like this

import org.apache.spark.sql.functions.dayofweek

date_format.withColumn("day_of_week", dayofweek(col("date")))

CodePudding user response：

You get your error because you named your first dataframe date_format, which is the same name as the Spark's built-in function you want to use. So when you call date_format, you're retrieving your dataframe instead of date_format built-in function.

To solve this, you should either rename your first dataframe:

val df_1 = df_filter.withColumn("date", to_date(col("pickup_datetime")))
val df_2 = df_1.withColumn("day_of_week", date_format(col("date"), "EEEE"))

Or ensure that you're calling right date_format by importing functions and then call functions.date_format when extracting day of week:

import org.apache.spark.sql.functions

val date_format = df_filter.withColumn("date", to_date(col("pickup_datetime")))
val df_2 = date_format.withColumn("day_of_week", functions.date_format(col("date"), "EEEE"))