how to webscrape the "key statistics" from morningstar.com to python and pandas-CodePudding

so the link of the data i want to webscrape is this one https://www.morningstar.com/stocks/xidx/smsm/valuation

and the data i want to scrape is this one down bellow

Pleaseee help mee :(

i would like to have the table in my jupyter notebook so i can use pandas and python to do my stock and investing analysis

CodePudding user response：

The page loads the data through an API endpoint that requires the API key if accessed directly. But you can simulate the request sent by the browser to get the same data in python.
Use inspector to find out the exact request, copy the CURL request and convert it to python using online tool
Here is the converted request. You can get the results in json with response.json

import requests

headers = {
    'authority': 'api-global.morningstar.com',
    'accept': '*/*',
    'accept-language': 'en-US,en;q=0.9',
    'apikey': 'lstzFDEOhfFNMLikKa0am9mgEKLBl49T',
    'origin': 'https://www.morningstar.com',
    'referer': 'https://www.morningstar.com/stocks/xidx/smsm/valuation',
    'sec-ch-ua': '"Not?A_Brand";v="8", "Chromium";v="108", "Google Chrome";v="108"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"',
    'sec-fetch-dest': 'empty',
    'sec-fetch-mode': 'cors',
    'sec-fetch-site': 'same-site',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
    'x-api-realtime-e': 'eyJlbmMiOiJBMTI4R0NNIiwiYWxnIjoiUlNBLU9BRVAifQ.X-h4zn65XpjG8cZnL3e6hj8LMbzupQBglHZce7tzu-c4utCtXQ2IYoLxdik04usYRhNo74AS_2crdjLnBc_J0lFEdAPzb_OBE7HjwfRaYeNhfXIDw74QCrFGqQ5n7AtllL-vTGnqmI1S9WJhSwnIBe_yRxuXGGbIttizI5FItYY.bB3WkiuoS1xzw78w.iTqTFVbxKo4NQQsNNlbkF4tg4GCfgqdRdQXN8zQU3QYhbHc-XDusH1jFii3-_-AIsqpHaP7ilG9aBxzoK7KPPfK3apcoMS6fDM3QLRSZzjkBoxWK75FtrQMAN5-LecdJk97xaXEciS0QqqBqNugoSPwoiZMazHX3rr7L5jPM-ecXN2uEjbSR0wfg-57iHAku8jvThz4mtGpMRAOil9iZaL6iRQ.o6tR6kuOQBhnpcsdTQeZWw',
    'x-api-requestid': 'cdbb5a73-9654-4b31-a845-32844eb44ca8',
    'x-sal-contenttype': 'e7FDDltrTy tA2HnLovvGL0LFMwT KkEptGju5wXVTU=',
}

params = {
    'languageId': 'en',
    'locale': 'en',
    'clientId': 'MDC',
    'component': 'sal-components-valuation',
    'version': '3.79.0',
}

response = requests.get(
    'https://api-global.morningstar.com/sal-service/v1/stock/valuation/v3/0P0000BPTU',
    params=params,
    headers=headers,
)