Pandas Concat returning "InvalidIndexError: Reindexing only valid with uniquely valued Index ob-CodePudding

I have two dataframes with the same headers. When I try to concatenate (pd.concat()) these two dfs (df1, df2), I get the error:

"InvalidIndexError: Reindexing only valid with uniquely valued Index objects"

I figured out that the problem is that there are duplicate column names within each dataframe. Example:

df1

respondent ID - Column1 - Column2 - Column3 - Column1 - Column2 - Column3

df2

respondent ID - Column1 - Column2 - Column3 - Column1 - Column2 - Column3

in theory, I just want to add the data of df2 below df1 (without the headers of df2 ofc)

How do I bypass this? Any thoughts?

CodePudding user response：

You can use this solution:

df1 = df1.append(df2, ignore_index=True)

CodePudding user response：

The duplicated columns shouldn't be an issue (even with ignore_index=False):

df1 = pd.DataFrame([range(7)], columns=['respondent ID', 'Column1', 'Column2', 'Column3', 'Column1', 'Column2', 'Column3'])
df2 = pd.DataFrame([['2']*7], columns=['respondent ID', 'Column1', 'Column2', 'Column3', 'Column1', 'Column2', 'Column3'])
pd.concat([df1, df2], ignore_index=True)

output:

  respondent ID Column1 Column2 Column3 Column1 Column2 Column3
0             0       1       2       3       4       5       6
1             2       2       2       2       2       2       2