Read multiple csv files from multiple folders in Python-CodePudding

I have a folder that includes folders and these folders include many csv files. I want to import and concatenate all of them in Python.

Let's say main folder: /main

subfolders: /main/main_1

csv: /main/main_1/first.csv

path='/main'
df_list = [] 
for file in os.listdir(path): 
    df = pd.read_csv(file) 
    df_list.append(df) 

final_df = df.append(df for df in df_list)

CodePudding user response：

Remember to add back main to the path:

df =pd.read_csv(path "/" file)

CodePudding user response：

I found a way to concat all of them but it doesn't satisfy to me as it takes too much time due to computational complexity.

path = "/main"

folders = []
directory = os.path.join(path)
for root,dirs,files in os.walk(directory):
    folders.append(root)

del folders[0]


final = []
for folder in folders:
    df = pd.concat(map(pd.read_csv, glob.glob(os.path.join(folder   "/*.csv"))))
    final.append(df)