I was given assistance with dedupe question I had to remove duplicate rows in databricks with Pyspark.

The dataframe that I execute the code on is as follows:

The Pyspark code is as follows:

The PySpark is as follows:

from pyspark.sql.window import Window
from pyspark.sql.functions import row_number

df2 = partdf.withColumn("rn", row_number().over(Window.partitionBy("Id","CreatedOn").orderBy("SinkModifiedOn")))
df3 = df2.filter("rn = 1").drop("rn")

After executing the code absolutely no rows are returned.

Can someone take a look at the code at let me know why after deduping no rows are returned?

My guess is that it has something to do with the SinkModified on field

CodePudding user response：

Use dropDuplicates to remove duplicate rows :

df = partdf.dropDuplicates(subset=['Id', 'createdon', 'SinkModifiedOn', 'caltype'])

Or

in this specific case, just simply:

partdf.drop_duplicates()