I'm trying to dynamically create a database without hard coding all of the API response values for columns and rows so I want to learn how to automatically parse through JSON and put all the keys/values in a variable so I can sort through and put them in the db.

Lets say I had some JSON like this:

(modified snippet from destiny 2 API response indented by 5)

{
     "Response": {
          "profile": {
               "data": {
                    "userInfo": {
                         "crossSaveOverride": 1,
                         "applicableMembershipTypes": [
                              3,
                              1
                         ],
                         "isPublic": true,
                         "membershipType": 1,
                         "membershipId": "123",      
                         "displayName": "Name1",
                         "bungieGlobalDisplayName": "name again",
                         "bungieGlobalDisplayNameCode": 0000
                    },
                    "dateLastPlayed": "2021-6-18T02:33:01Z",        
                    "versionsOwned": 127,
                    "characterIds": [
                         "id1",
                         "id2",
                         "id3"
                    ],
                    "seasonHashes": [
                         3243434344,
                         3423434244,
                         3432342443,
                         3434243244
                    ],
                    "currentSeasonHash": 3434243244,
                    "currentSeasonRewardPowerCap": 1330
               },
               "privacy": 1
          }
     },
     "ErrorCode": 1,
     "ThrottleSeconds": 0,
     "ErrorStatus": "Success",
     "Message": "Ok",
     "MessageData": {}
}

I want to automatically parse through and get all of the key/values minus the error code area so everything under "Response". It would all go into the database for example:

I know how to parse normally or loop through but only to grab values like:

displayName = Response["Response"]["profile"]["data"]["userInfo"]["displayName"]

How does one grab all keys and values and then automatically store in a variable from the lowest level? Also, how would one exclude keys if they are not needed?

EDIT: Adding Clarification

I learned that this JSON response type is a dict and I can use Response.keys() and Response.values() to get the keys and values.

What I am asking is, from Response["Response"], how to get all of the keys and values down to the bottom level and exclude the ones I don't need.

For example if I do:

r = Response["Response"] 
for key in r.keys():
    print(key)

I'm only going to get profile which is what I don't need.

I would have to do:

r = Response["Response"]["profile"]["data"]["userInfo"] 
for key in r.keys():
    print(key)

which would return

crossSaveOverride
applicableMembershipTypes
isPublic
membershipType
membershipId
displayName
bungieGlobalDisplayName
bungieGlobalDisplayNameCode

Which is what I need but I do not want to manually define ["Response"]["profile"]["data"]["userInfo"]or similar for every response. I want to grab everything automatically while excluding the items I don't need.

CodePudding user response：

Dmitry Torba over at Get all keys of a nested dictionary posted a function that does what you want.

def recursive_items(dictionary):
    for key, value in dictionary.items():
        if type(value) is dict:
            yield from recursive_items(value)
        else:
            yield (key, value)

a = {'a': {1: {1: 2, 3: 4}, 2: {5: 6}}}

for key, value in recursive_items(a):
    print(key, value)