How to merge all csv files in a folder, but they all have different names-CodePudding

I've been searching for a way to merge all csv files in a folder. They all have the same headers, but different names. I've found some videos on youtube on merge and some questions here on stackoverflow that touches the matter. The problem is that this tutorials are focused on files with the same name as: sales1, sales2, etc.

In my case, all files in the directory are CSVs and are located in 'D:\XXXX\XXXX\output'

The code I have used is:

import pandas as pd

# set files path
amazon = r'D:\XXXX\XXXX\output\amazonbooks.csv'
bookcrossing = r'D:\XXXX\XXXX\output\bookcrossing.csv'

# merge files
dataFrame = pd.concat(
   map(pd.read_csv, [amazon, bookcrossing]), ignore_index=True)
print(dataFrame)

If the code could merge all the files that stand in the folder output (since all of them are .csv), instead of naming each one of them, it would be better.

I'd be glad if anyone can help me with this problem, or can guide me on how to solve this.

CodePudding user response：

If the goal is to append the files into a single result, you don't really need any CSV processing at all. Just write the file contents minus the header line (except the first one). glob will return file names with path that match the pattern, "*.csv".

from glob import glob
import os
import shutil

csv_dir = r'D:\XXXX\XXXX\output'
result_csv = r'd:\XXXX\XXXX\combined.csv'
first_hdr = True

# all .csv files in the directory have the same header
with open(result_csv, "w", newline="") as result_file:
    for filename in glob(os.path.join(csv_dir, "*.csv")):
        with open(filename) as in_file:
            header = in_file.readline()
            if first_hdr:
                result_file.write(header)
                first_hdr = False
            shutil.copyfileobj(in_file, result_file)

CodePudding user response：

(assuming all csvs have equal number of columns)
Try something like this:

import os
import pandas as pd

csvs = [file for file in os.listdir('D:\XXXX\XXXX\output\') if file.endswith('.csv')]

result_df = pd.concat([pd.read_csv(f'D:\XXXX\XXXX\output\{file}') for file in csvs])