When I try to get the source code for a website, I would usually use urllib.request.urlopen(url). However, when I try to do it for a website with a table (that I believe generated after javascript code is run), it does not return the same as what I see when I press inspect on a website.

How can I get what I see in inspect after the page is loaded?

I have tried using json backage and load the page using json.load(urllib.request.urlopen(url)), I would get an error:

I also tried to use selenium Webdriver, but when I call webdrive.Chrome() an error is generated:

WebDriverException: Message: Service chromedriver unexpectedly exited. Status code was: 127

How can I fix either of these solutions? Or is there a better solution? Any help would be appeciated.

I'm using replit if that matters.

CodePudding user response：

From what I can see in the network logs, the site seems to be using an API to load the contents of the table. If you copy the API URL and break down the query:

# from urllib.parse import urlsplit, parse_qsl
apiUrl = 'https://datacenter-web.eastmoney.com/api/data/v1/get?callback=jQuery112309834864079678796_1673633495739&sortColumns=SECURITY_CODE,TRADE_DATE&sortTypes=1,-1&pageSize=50&pageNumber=1&reportName=RPT_DAILYBILLBOARD_DETAILSNEW&columns=SECURITY_CODE,SECUCODE,SECURITY_NAME_ABBR,TRADE_DATE,EXPLAIN,CLOSE_PRICE,CHANGE_RATE,BILLBOARD_NET_AMT,BILLBOARD_BUY_AMT,BILLBOARD_SELL_AMT,BILLBOARD_DEAL_AMT,ACCUM_AMOUNT,DEAL_NET_RATIO,DEAL_AMOUNT_RATIO,TURNOVERRATE,FREE_MARKET_CAP,EXPLANATION,D1_CLOSE_ADJCHRATE,D2_CLOSE_ADJCHRATE,D5_CLOSE_ADJCHRATE,D10_CLOSE_ADJCHRATE,SECURITY_TYPE_CODE&source=WEB&client=WEB&filter=(TRADE_DATE<='2023-01-13')(TRADE_DATE>='2023-01-13')'
dict(parse_qsl(urlsplit(url).query))

you can see the parameters

{'callback': 'jQuery112309834864079678796_1673633495739',
 'sortColumns': 'SECURITY_CODE,TRADE_DATE',
 'sortTypes': '1,-1',
 'pageSize': '50',
 'pageNumber': '1',
 'reportName': 'RPT_DAILYBILLBOARD_DETAILSNEW',
 'columns': 'SECURITY_CODE,SECUCODE,SECURITY_NAME_ABBR,TRADE_DATE,EXPLAIN,CLOSE_PRICE,CHANGE_RATE,BILLBOARD_NET_AMT,BILLBOARD_BUY_AMT,BILLBOARD_SELL_AMT,BILLBOARD_DEAL_AMT,ACCUM_AMOUNT,DEAL_NET_RATIO,DEAL_AMOUNT_RATIO,TURNOVERRATE,FREE_MARKET_CAP,EXPLANATION,D1_CLOSE_ADJCHRATE,D2_CLOSE_ADJCHRATE,D5_CLOSE_ADJCHRATE,D10_CLOSE_ADJCHRATE,SECURITY_TYPE_CODE',
 'source': 'WEB',
 'client': 'WEB',
 'filter': "(TRADE_DATE<='2023-01-13')(TRADE_DATE>='2023-01-13')"}

^{(Btw, from looking the request's initiator, the tradedetail script seems to be mostly responsible for generating the parameters.)}

You can reform the link

start, end, pgSize, pgNum = '2023-01-13', '2023-01-13', 50, 1
auQstr = 'sortColumns=SECURITY_CODE,TRADE_DATE&sortTypes=1,-1'
auQstr  = f'&pageSize={pgSize}&pageNumber={pgNum}' ## max pageSize seems to be 500
auQstr  = '&reportName=RPT_DAILYBILLBOARD_DETAILSNEW&columns=SECURITY_CODE,SECUCODE,SECURITY_NAME_ABBR,TRADE_DATE,EXPLAIN,CLOSE_PRICE,CHANGE_RATE,BILLBOARD_NET_AMT,BILLBOARD_BUY_AMT,BILLBOARD_SELL_AMT,BILLBOARD_DEAL_AMT,ACCUM_AMOUNT,DEAL_NET_RATIO,DEAL_AMOUNT_RATIO,TURNOVERRATE,FREE_MARKET_CAP,EXPLANATION,D1_CLOSE_ADJCHRATE,D2_CLOSE_ADJCHRATE,D5_CLOSE_ADJCHRATE,D10_CLOSE_ADJCHRATE,SECURITY_TYPE_CODE&source=WEB&client=WEB'
auQstr  = f'&filter=(TRADE_DATE<='{end}')(TRADE_DATE>='{start}')'
apiUrl = f'https://datacenter-web.eastmoney.com/api/data/v1/get?{auQstr}'

and retrieve table data with

# import requests, json, pandas

apiReq = requests.get(apiUrl)
# print(apiReq.status_code, apiReq.reason, 'from', apiReq.url)
# apiReq.raise_for_status()

jDict = {}
try: jDict = json.loads(apiReq.text.strip().strip('jQuery_(0123456789);'))
except: print('failed to extract JSON from', apiReq.text)
rDict =  jDict.get('result') if hasattr(jDict.get('result', {}), 'get') else {}
# for k, v in jDict.items(): print(f'{k}:', type(v) if hasattr(v, 'pop') else v)
# for k, v in rDict.items(): print(f'result_{k}:', type(v) if hasattr(v, 'pop') else v)

[I couldn't quite figure out how to generate the callback parameter, but I don't see any difference in the data returned either way except that with the callback, the response is wrapped like {callback}({JSON}); (that's why I added the .strip('jQuery_(0123456789);') part).]

Now pandas.DataFrame(rDict.get('data', [])) should return a DataFrame that looks like

^{The table above was printed with print(pandas.DataFrame(rDict.get('data', [])).to_markdown(disable_numparse=True)).}