I'm working on JSON data from this API call: https://api.nfz.gov.pl/app-umw-api/agreements?year=2022&branch=01&productCode=01.0010.094.01&page=1&limit=10&format=json&api-version=1.2

This is page 1, but there are 49 pages in total, therefore a part of my code deals (successfully) with pagination. I don't want to save this JSON in a file and, if I can avoid it, don't really want to import the 'json' package - but will do if necessary.

A variation of this code works correctly if I'm pulling entire ['data']['agreements'] dictionary (or is it a list...). But I don't want that, I want individual parameters for all the 'attributes' of each 'agreement'. In my code below I'm trying to pull the 'provider-name' attribute, and would like to get a list of all the provider names, without any other data there.

But I keep getting the "list indices must be integers or slices, not str" error in line 18. I've tried many ways to get this data which is nested within a list nested within a dictionary, etc. like splitting it further into another 'for' loop, but no success.

import requests
import math
import pandas as pd


baseurl = 'https://api.nfz.gov.pl/app-umw-api/agreements?year=2022&branch=01&productCode=01.0010.094.01&page=1&limit=10&format=json&api-version=1.2'

def main_request(baseurl, x):
    r = requests.get(baseurl   f'&page={x}')
    return r.json()

def get_pages(response):
    return math.ceil(response['meta']['count'] / 10)

def get_names(response):
    providerlist = []
    all_data = response['data']['agreements']
    for attributes1 in all_data ['data']['agreements']:
        item = attributes1['attributes']['provider-name']
        providers = {
            'page1': item,
        }

    providerlist.append(providers)
    return providerlist

mainlist = []
data = main_request(baseurl, 1)
for x in range(1,get_pages(data) 1):
    mainlist.extend(get_names(main_request(baseurl, x)))

mydataframe = pd.DataFrame(mainlist)

print(mydataframe)

CodePudding user response：

To get the data from the Json to the dataframe you can use next example:

import requests
import pandas as pd


api_url = "https://api.nfz.gov.pl/app-umw-api/agreements?year=2022&branch=01&productCode=01.0010.094.01&page={}&limit=10&format=json&api-version=1.2"

all_data = []
for page in range(1, 5): # <-- increase page numbers here
    data = requests.get(api_url.format(page)).json()

    for a in data["data"]["agreements"]:
        all_data.append({"id": a["id"], **a["attributes"], "link": a["links"]['related']})

df = pd.DataFrame(all_data)
print(df.head().to_markdown(index=False))

Prints: