Extracting Year,Month And Hour from a column using Spark Scala-CodePudding

I have a data file in which a column datetime has value 01-01-2011 00:00. I want to extract year, month and hour from this value. Code:

val actualData=df1.select(year(col("datetime")).as("Year"),
month(col("datetime")).as("Month"),
hour(col("datetime")).as("Hour"),
dayofmonth(col("datetime")).as("DayOfMonth"))

Output i am getting:

 --------------------- ---- ----- ---- ---------- 
Year|Month|Hour|DayOfMonth|
 --------------------- ---- ----- ---- ---------- 
|null| null|null| null|
|null| null|null| null|
|null| null|null| null|
|null| null|null| null|

CodePudding user response：

Seems column datetime is not in appropriate datatype. Casting the datetime column to date should help.

    df1 = df1.withColumn("datetime",to_date(col("datetime"),"dd-MM-yyyy HH:mm"))

CodePudding user response：

The format of "datetime" is not "yyyy-MM-dd HH:mm", so it need to specify the format of it.

val actualData=df1.select(year(to_date(col("datetime"), "dd-MM-yyyy HH:mm")).as("Year"),
month(to_date(col("datetime"), "dd-MM-yyyy HH:mm")).as("Month"),
hour(to_date(col("datetime"), "dd-MM-yyyy HH:mm")).as("Hour"),
dayofmonth(to_date(col("datetime"), "dd-MM-yyyy HH:mm")).as("DayOfMonth"))