Getting different HTML code from python request.get() and chrome inspect element while using same UR-CodePudding

I am trying to scrape data from this

CodePudding user response：

If you turn off JavaScript then click on position, it wouldn't work but if you turn on JS and click on position then it will work. So you are not getting the the position data because that data is loaded dynamically by JS via API as POST method.

Example:

import requests
import json
api_url ='https://www.binance.com/bapi/futures/v1/public/future/leaderboard/getOtherPosition'

headers= {
    "content-type":"application/json",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"}


body = {"encryptedUid":"D64DDD2177FA081E3F361F70C703A562","tradeType":"PERPETUAL"}

res = requests.post(api_url,headers=headers,data=json.dumps(body)).json()
data =[]
for item in res['data']['otherPositionRetList']:
    data.append({
        'symbol':item['symbol'],
        'entryPrice':item['entryPrice'],
        'markPrice':item['markPrice']
        })
print(data)

Output:

[{'symbol': 'LUNA2USDT', 'entryPrice': 2.329198591774, 'markPrice': 2.44419197}, {'symbol': '1000LUNCBUSD', 'entryPrice': 0.2868522607698, 'markPrice': 0.27523487}]