How to format month in full in pyspark?-CodePudding

what's the best way to format a date, with the month in full, for example, I get the value 01-01-2022 and I need to return 01JAN2022, I tried the following code but it didn't work:

.withColumn('dt_ori',upper(date_format(col("dt_ori"),"ddMMMyyyy")))

CodePudding user response：

This is working as expected , based on the various Date Time patterns available within Spark 3.x

'MMM': Short textual representation in the standard form. The month pattern should be a part of a date pattern not just a stand-alone month except locales where there is no difference between stand and stand-alone forms like in English.

Spark Version

Spark Version - 3.1.2
Java Version - 1.8.0_292 (AdoptOpenJDK)
Scala Version - 2.12.10

Date Format

s = StringIO("""
ID,date_str
101,01-01-2022
""")

df = pd.read_csv(s,delimiter=',')

sparkDF = sql.createDataFrame(df)\
            .withColumn('date_parsed',F.to_date(F.col('date_str'), 'dd-MM-yyyy'))\
            .withColumn('date_formated',F.upper(F.date_format(F.col('date_parsed'), 'ddMMMyyyy')))\
            .drop('date_str')

sparkDF.show()

 --- ----------- ------------- 
| ID|date_parsed|date_formated|
 --- ----------- ------------- 
|101| 2022-01-01|    01Jan2022|
 --- ----------- -------------