Not able to fetch bs4 table contents in python-CodePudding

I want to fetch all the user handles present in this link

Any solution for this problem will be will be highly appreciated.

CodePudding user response：

The url is dynamic and beautifulsoup can't render JavaScript but Data is generating from API meaning the website is using API.

import requests
api_url='https://practiceapi.geeksforgeeks.org/api/v1/leaderboard/ranking/?ranking_type=overall&page={page}'
for page in range(1,11):
    data=requests.get(api_url.format(page=page)).json()
    for handle in data:
        print(handle['user_handle'])

Output:

Ibrahim Nash
blackshadows
mb1973
Quandray
akhayrutdinov
saiujwal13083
shivendr7
kirtidee18
mantu_singh
cfwong8
harshvardhancse1934
sgupta9519
sanjay05
samiranroy0407
Maverick_H
sreerammuthyam999
gfgaccount
sushant_a
verma_ji
balkar81199
marius_valentin_dragoi
ishu2001mitra
_tony_stark_01
ta7anas17113011638
yups0608
himanshujainmalpura
yujjwal9700
parthabhunia_04
KshamaGupta
the_coder95
ayush_gupta4
khushbooguptaciv18
aditya dhiman
dilipsuthar00786
adityajain9560
dharmsharma0811
Aegon_Targeryan
1032180422
mangeshagarwal1974
naveedaamir484
raj_271
Pulkit__Sharma__
aroranayan999
surbhi_7
ruchika1004ajmera
cs845418
shadymasum
lonewolf13325
user_1_4_13_19_22
SubhankarMajumdar

CodePudding user response：

You can get this using Selenium.

from selenium import webdriver

driver = webdriver.Chrome(executable_path = "<webdriver path>")
driver.get("https://practice.geeksforgeeks.org/leaderboard/")

user_names = driver.find_elements(by = "xpath", value = "//tbody[@id = 'overall_ranking']/tr/td/a")
user_names = list(map(lambda name:name.text, user_names))

driver.quit()