I would like to select the dataframe which takes the highest values ​​by Matricule. For exemple in this table.

we obtain This table :

Please can we help me. Thanks.

CodePudding user response：

The way I used to solve your problem is to create a maxValue column with Function Max and partition over Matricule column.

You can find the code below:

from pyspark.sql import functions as F
from pyspark.sql import Window as W

(df
  .withColumn('maxValue', F.max(F.col('Value')).over(W.partitionBy(F.col('Matricule'))))
  .filter(F.col('Value') == F.col('maxValue'))
  .drop('maxValue')
)