I'm trying to read 100 CSVs and collate data from all into a single CSV. I made use of :

all_files = pd.DataFrame()
for file in files :
   all_files = all_files.append(pd.read_csv(file,encoding= 'unicode_escape')).reset_index(drop=True)

where files = list of filepaths of 100 CSVs

Now each CSV may have different number of columns. single CSV, each row may have different no. of colums too. I want to match the column headers names, put the data from all the CSVs in the correct column, and keep on adding new columns to my final DF on the go.

The above code works fine for 30-40 CSVs and then breaks and gives the following error:

ParserError: Error tokenizing data. C error: Expected 16 fields in line 78, saw 17

Any help will be much appreciated!

CodePudding user response：

There are a couple of ways to read variable length csv files -

First, you can specify the column names beforehand. If you are not sure of the number of columns, you can give a reasonably large number of columns

df = pd.read_csv(filename.csv, header=None, names=list(range(10)))

The other option is to read the entire file into a single column using a different delimiter - and then split on commas

df = pd.read_csv(filename.csv, header=None, sep='\t')
df = df[0].str.split(',', expand=True)

CodePudding user response：

Its because you are trying to read all CSV files into a single Dataframe. When the first file is read number of columns for the DataFrame are decided and then it results in error when a different number of columns are fed. If you really want to concat them you should read them all in python, adjust their coulmns and then concat them