Function to take a list of spark dataframe and convert to pandas then csv-CodePudding

import pyspark
            
dfs=[df1,df2,df3,df4,df5,df6,df7,df8,df9,df10,df1,df12,df13,df14,df15]
    
for x in dfs:
    y=x.toPandas()
    y.to_csv("D:/data")

This is what I wrote, but I actually want the function to take this list and convert every df into a pandas df and then convert it to csv and save it in the order as it appears on dfs list and save it to a particular directory in the order of name. Is there a possible way to write such function? PS D:/data is just an imaginary path and is used for explanation.

CodePudding user response：

If you will convert a dataframe to a csv, you still need to state it in df.to_csv. So, try:

for x in dfs:
       y=x.toPandas()
       y.to_csv(f"D:/data/df{dfs.index(x)   1}.csv")

I set it as df{dfs.index(x) 1} so that the file names will be df1, df2, ... etc.