I am trying to access the data on every page that exists on the site at

url = 'https://apexranked.com/'

page = 1 

while page != 121: 
    url = f'https://apexranked.com/?page={page}'
    print(url) 
    page = page   1

CodePudding user response：

That website uses Javascript to fetch pages. You can investigate the urls fetched by JS and try to follow them, or you can use Selenium to scrape it.

CodePudding user response：

You can use this example how to navigate the pages and load the data into pandas DataFrame:

import requests
import pandas as pd


url = "https://apexranked.com/wp-admin/admin-ajax.php"

params = {
    "action": "get_player_data",
    "page": "2",
    "total_pages": "196",
}

all_df = []
for params["page"] in range(1, 3):  # <-- increase number of pages here
    df = pd.read_html(requests.get(url, params=params).text)[0]
    all_df.append(df)

final_df = pd.concat(all_df)
print(final_df.tail(10).to_markdown(index=False))

Prints: