Reading multiple CSV files from path-CodePudding

I have a set of CSV files that I want to read them from a Windows path. My code is this:

directory = os.listdir(r'C:\Users\User\Documents\etc...')

for files in directory:
    print(files)
    dataset = pd.read_csv(files, header = None)
    trainSet = dataset.values.tolist()
    editedSet = dataset.values.tolist()

The problem with this is that I get error No such file or directory: "filename" while the files of course exist in the directory. The solutions I have found are for plain text files while I have CSVs. Any suggestion on what I am doing wrong?

Thank you

CodePudding user response：

os.listdir returns list of files inside the directory relative to the input path

You need to join it with the input path to actually calls the file

base_dir = r'C:\Users\User\Documents\etc...'
directory = os.listdir(base_dir)

for files in directory:
    print(files)
    path = os.path.join(base_dir, files)
    print(path)
    dataset = pd.read_csv(path, header = None)
    trainSet = dataset.values.tolist()
    editedSet = dataset.values.tolist()

CodePudding user response：

The problem is that you are only providing filenames as the argument. You should be using a "path" instead.

There are many ways to solve this issue. A modern approach is to use pathlib instead of os for file management.

This is a solution with pathlib.

from pathlib import Path

root_path = Path(r'C:\Users\User\Documents\etc...')
file_paths = root_path.glob('*.csv')
files = [root_path.joinpath(file_path) for file_path in file_paths]

for file in files:
    print(file)
    dataset = pd.read_csv(file, header = None)
    trainSet = dataset.values.tolist()
    editedSet = dataset.values.tolist()