do you know any way how I can overwrite certain partitions in a folder, and leave other partitions unchanged?

CodePudding user response：

If your dataframe contains only data from the partitions you would like to overwrite then you need to set spark.sql.sources.partitionOverwriteMode to dynamic (Take a look here as well).