Pulling paginated json data with python / loop-CodePudding

I pulled json data with api. Response example as below.

{
  "page": 1,
  "pages": 5,
  "limit": 100,
  "count": 466,
  "records": [
    {
      "epoch": "9b4WLoXXYgga9mGRLGsWKu",
      "last_height": 5402,
      "last_time": "2021-02-22T01:02:43.011692Z",
      "expected_blocks": 432,
      "produced_blocks": 432,
    }
  ]
}

But page is limited with 100 rows. Parameters query is available for pages. Below is python code I use:

import requests
import json
import pandas as pd

response = requests.get(url, params = {'page': 1})
json_data = json.loads(response.text)
df = pd.DataFrame(json_data['records']) 
print(df)

How can I get all pages with looping?

Thanks

CodePudding user response：

You can try this and get all your data in dataframe by looping.

import requests
import json
import pandas as pd

response = requests.get(url, params = {'page': 1}) #---> remember to keep your url 
records = []
for page_number in range(2, response.json().get("pages") 1):
    response = requests.get(url, params = {'page': page_number})
    records  = response.json().get('records')
df = pd.DataFrame(records) 
print(df)