I would like to save the whole Azure Prices REST API to CSV.

In order to do so I have to query the endpoint

I wrote a Python scripts that could help me grab that NextPageLink and loop it into a function:

import requests
import json
import pandas as pd 
from timeit import default_timer as timer
from datetime import timedelta

start = timer()
NextPageLink = "https://prices.azure.com/api/retail/prices"

def GetJSON(NextPageLink):
    wjdata = requests.get(NextPageLink).json()
    df = pd.DataFrame(wjdata)
    df.to_csv("test.csv", index=False)
    if 'NextPageLink' in wjdata:
        print (timer(), wjdata['NextPageLink'])
        NextPageLink = wjdata['NextPageLink']
        return NextPageLink

GetJSON(NextPageLink) 

The script is quite simple but it just saves the first page and doesn't query the NextPageLink.

What am I doing wrong?

CodePudding user response：

To get all data from the API you can try (there are 4558 requests total):

import requests
import pandas as pd

url = "https://prices.azure.com/api/retail/prices"

all_data = []
while True:
    print(url)
    data = requests.get(url).json()
    all_data.extend(data["Items"])
    if data["NextPageLink"]:
        url = data["NextPageLink"]
    else:
        break

df = pd.DataFrame(all_data)
print(df.head().to_markdown(index=False))
df.to_csv("data.csv", index=False)

Prints:

......

https://prices.azure.com:443/api/retail/prices?$skip=455600
https://prices.azure.com:443/api/retail/prices?$skip=455700
https://prices.azure.com:443/api/retail/prices?$skip=455800

and saves data.csv (screenshot from LibreOffice):