Is it possible to clear the cache in Azure Databricks Notebook? I converted a column from string to date and I see it was really converted from the job run, like so:

df.withColumn('Date', F.to_date('Date', 'yyyyMMdd'))
Out[167]: DataFrame[bla bla bla, Date: date]

But when I check the dataframe schema, I am still seeing string

CodePudding user response：

Just restart the kernel and you should be good to go.

CodePudding user response：

Probably, you just need to assign your transformation.

df = df.withColumn('Date', F.to_date('Date', 'yyyyMMdd'))

Any transformation you make on dataframe are not "inplace". They return a new dataframe that you need to assign to a variable to save the transformation.