I want to convert a list of dictionaries to a pandas dataframe, however, I got ValueError: DataFrame constructor not properly called!

Below is an example and how I got the data:

import requests
import pandas as pd

# Send an HTTP GET request to the URL
response = requests.get(url)

# Decode the JSON data into a dictionary
scrapped_data = response.text

scraped_data = [{"id":123456,"date":"12-12-2022","value":37}, {"id":123456,"date":"13-12-2022","value":38}]

I want to convert it to a dataframe format like the following:

I tried the following methods:

df = pd.DataFrame(scrapped_data)

df = pd.DataFrame_from_dict(scrapped_data)

df = pd.DataFrame(scrapped_data, orient='columns')

all got the same value errors.

I also tried:

df = pd.json_normalize(scrapped_data)

but got NotImplementedError

The type for scrapped_data is string format

Thanks for your help, let me know if you have any questions

CodePudding user response：

One reason for receiving this error from pandas is providing str as data. I think your data come as str, If it is the case then Try this:

import json
import pandas as pd

orignal_data='[{"id":"123456","date":"12-12-2022","value":"37"}, {"id":"123456","date":"13-12-2022","value":"38"}]'
scraped_data = json.loads(orignal_data)
df = pd.DataFrame(data=scraped_data)
df

CodePudding user response：

As you said, scrapped_data is a string then you need to convert it into a dictionary (with the method loads from the json library for example).

If scrapped_data = '[{"id":"123456","date":"12-12-2022","value":"37"}, {"id":"123456","date":"13-12-2022","value":"38"}]', then you can just do df = pd.DataFrame(scrapped_data).