How to read multiple csv from folder without concatenating each file-CodePudding

I have a folder and inside the folder suppose there are 1000 of .csv files stored. Now I have to create a data frame based on 50 of these files so instead of loading line by line is there any fast approach available?

And I also want the file_name to be the name of my data frame?

I tried below method but it is not working.

# List of file that I want to load out of 1000
path = "..."
file_names = ['a.csv', 'b.csv', 'c.csv', 'd.csv', 'e.csv']

for i in range(0, len(file_names)):
    col_names[i] = pd.read_csv(path   col_name[i])

But when I tried to read the variable name it is not displaying any result. Is there any way I can achieve the desired result.

I have checked various article but in each of these article at the end all the data has been concatenated and I want each file to be loaded indiviually.

CodePudding user response：

IIUC, what you want to do is create multiple dfs, and not concatenate them in this case well you can read it using read_csv, and them stackings the return df objects in a list

your_paths = [# Paths to all of your wanted csvs]
l = [pd.read_csv(i) for i in your_paths] # This will give you a list of your dfs
l[0] # One of your dfs

If you want them named, you can make it as dict with different named keys

You can access them individually, through index slicing or key slicing depends on the data structure you use.

Would not recommend this action tho, as well it is counter intuitive and well multiple df objects use a little more memory than a unique one

CodePudding user response：

file_names = ['a.csv', 'b.csv', 'c.csv', 'd.csv', 'e.csv']
data_frames = {}
for file_name in file_names:
    df  = pd.read_csv(file_name)
    data_frames[file_name.split('.')[0]] = df

Now you can reach any data frame from data_frames dictionary; as data_frames['a'] to access a.csv

CodePudding user response：

try:

import glob

p = glob.glob( 'folder_path_where_csv_files_stored/*.csv' ) #1. will return a list of all csv files in this folder, no need to tape them one by one.
d = [pd.read_csv(i) for i in p] #2. will create a list of dataframes: one dataframe from each csv file
df = pd.concat(d, axis=0, ignore_index=True) #3. will create one dataframe `df` from those dataframes in the list `d`