How can I fix/ workaround this KeyError when I try to extract from a json via my python API request?-CodePudding

I have been trying to seed a django DB with some covid data from an api and get a KeyError for a particular data type - in the source it is a floating_timstamp ("lab_report_date" : "2014-10-13T00:00:00.000"). (edit: not sure if the type is relevant, but trying to be comprehensive here).

I tried doing a more simple API request in python but get the same keyError. Below is my code and the error message.

import requests
response = requests.get("https://data.cityofchicago.org/resource/naz8-j4nc.json")
print(response.json())

The output looks like this:

[
    {
        "cases_age_0_17": "1",
        "cases_age_18_29": "1",
        "cases_age_30_39": "0",
        "cases_age_40_49": "1",
        "cases_age_50_59": "0",
        "cases_age_60_69": "0",
        "cases_age_70_79": "1",
        "cases_age_80_": "0",
        "cases_age_unknown": "0",
        "cases_asian_non_latinx": "1",
        "cases_black_non_latinx": "0",
        "cases_female": "1",
        "cases_latinx": "1",
        "cases_male": "3",
        "cases_other_non_latinx": "0",
        "cases_total": "4",
        "cases_unknown_gender": "0",
        "cases_unknown_race_eth": "1",
        "cases_white_non_latinx": "1",
        "deaths_0_17_yrs": "0",
        "deaths_18_29_yrs": "0",
        "deaths_30_39_yrs": "0",
        "deaths_40_49_yrs": "0",
show more (open the raw output data in a text editor) ...

        "hospitalizations_unknown_gender": "3",
        "hospitalizations_unknown_race_ethnicity": "16",
        "hospitalizations_white_non_latinx": "135"
    }
]

So far so good, but if I try to extract the problem key, i get the KeyError:

report_date = []
for i in response.json():
    ls = i['lab_report_date']
report_date.append(ls)

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/var/folders/h3/5wlbmz0s3jb978hyhtvf9f4h0000gn/T/ipykernel_2163/2095152945.py in <module>
      1 report_date = []
      2 for i in response.json():
----> 3     ls = i['lab_report_date']
      4 report_date.append(ls)

KeyError: 'lab_report_date'

This issue occurs with or without using a for loop. I've gotten myself real turned around, so apologies if there are any errors or omissions in my code.

CodePudding user response：

Because there's an item in the array response.json() that does not contain a key lab_report_date. That happens when the backend data is not so clean.

So what you need to do is to use try-except code block to handle this exception. The following code runs well now.

import requests
response = requests.get("https://data.cityofchicago.org/resource/naz8-j4nc.json")
print("The total length of response is %s" % len(response.json()))

report_date = []
for i in response.json():
    try:
        ls = i['lab_report_date']
        report_date.append(ls)
    except:
        print("There is an item in the response containing no key lab_report_date:")
        print(i)

print("The length of report_date is %s" % len(report_date))

The output of the above code is as follows.

The total length of response is 592
There is an item in the response containing no key lab_report_date:
{'cases_total': '504', 'deaths_total': '1', 'hospitalizations_total': '654', 'cases_age_0_17': '28', 'cases_age_18_29': '116', 'cases_age_30_39': '105', 'cases_age_40_49': '83', 'cases_age_50_59': '72', 'cases_age_60_69': '61', 'cases_age_70_79': '25', 'cases_age_80_': '14', 'cases_age_unknown': '0', 'cases_female': '264', 'cases_male': '233', 'cases_unknown_gender': '7', 'cases_latinx': '122', 'cases_asian_non_latinx': '15', 'cases_black_non_latinx': '116', 'cases_white_non_latinx': '122', 'cases_other_non_latinx': '30', 'cases_unknown_race_eth': '99', 'deaths_0_17_yrs': '0', 'deaths_18_29_yrs': '0', 'deaths_30_39_yrs': '0', 'deaths_40_49_yrs': '1', 'deaths_50_59_yrs': '0', 'deaths_60_69_yrs': '0', 'deaths_70_79_yrs': '0', 'deaths_80_yrs': '0', 'deaths_unknown_age': '0', 'deaths_female': '0', 'deaths_male': '1', 'deaths_unknown_gender': '0', 'deaths_latinx': '0', 'deaths_asian_non_latinx': '0', 'deaths_black_non_latinx': '0', 'deaths_white_non_latinx': '1', 'deaths_other_non_latinx': '0', 'deaths_unknown_race_eth': '0', 'hospitalizations_age_0_17': '30', 'hospitalizations_age_18_29': '78', 'hospitalizations_age_30_39': '74', 'hospitalizations_age_40_49': '96', 'hospitalizations_age_50_59': '105', 'hospitalizations_age_60_69': '111', 'hospitalizations_age_70_79': '89', 'hospitalizations_age_80_': '71', 'hospitalizations_age_unknown': '0', 'hospitalizations_female': '310', 'hospitalizations_male': '341', 'hospitalizations_unknown_gender': '3', 'hospitalizations_latinx': '216', 'hospitalizations_asian_non_latinx': '48', 'hospitalizations_black_non_latinx': '208', 'hospitalizations_white_non_latinx': '135', 'hospitalizations_other_race_non_latinx': '31', 'hospitalizations_unknown_race_ethnicity': '16'}
The length of report_date is 591