for loop in Pandas on returning limited items from list-CodePudding

I am reading in a JSON file contains all of the information coming from an API request. The file isn't very large, only about 200 items. I am attempting to loop through each item, store it as a pandas DataFrame, append it to a list, and concat the results into one DataFrame.

df_list = []
list_length = 53
for i in range(list_length):
  df = pd.DataFrame(contenders_list[i]).T.reset_index()
  df_list.append(df)

new_df = pd.concat(mylist)
new_df.head()

If I run this, it works. I have a DataFrame with the first 53 items from the JSON file. However, if I go above 53, like the actual length of the list, I get the following error:

ValueError: If using all scalar values, you must pass in an index

Can anyone explain this?

CodePudding user response：

Have you tried loading the JSON as a pandas DataFrame yet? Pandas has a method called read_json, it converts your JSON into a pandas DF automatically. Have a look:

from pandas import read_json

# If the files are in the same folder, then you can simply write 'filename.json'
dataframe_from_json = read_json('path/to/your.json')

CodePudding user response：

You are not using the best method to create a Dataframe from a dictionary. Pandas is expecting the data values to be list values or dict values. According to the error raised, you are passing a scalar value. In that case, you also have to pass in the index.

let's say you do :

my_dict = {
  'a':'1',
  'b':'2',
  'c':'3'
}

# Convert dict to dataframe
df = pd.DataFrame.from_dict(my_dict)
df

you should indeed receive the following error:

ValueError: If using all scalar values, you must pass an index

But if you do :

my_dict = {
  'a':['1'],
  'b':['2'],
  'c':['3']
}

# Convert dict to dataframe
df = pd.DataFrame.from_dict(my_dict)
df

It will not raise any error and output as follows :

    a   b   c
0   1   2   3