I have a column in a dataframe that looks like "20221026 220032". I need to convert this to a datetime column but I don't have the colon separators for the time portion. I've attempted the following:

df_meta = df_meta.withColumn("filedatetime",F.to_timestamp(F.col("filedatetime_str"),"YYYYddmm HHmmss"))

But I get an error that it is an unrecognizable format. The reason my column comes this way is because is was parsed from a filename. Is there a way to convert this to a datetime without having to insert the colons?

CodePudding user response：

Try:

df_meta = df_meta.withColumn("filedatetime",F.to_timestamp(F.col("filedatetime_str"),"yMMdd HHmmss"))

spark date and time format can refer to Datetime Patterns