Scrape a JavaScript heavy website with requests bs4-CodePudding

I'm trying to scrape some data on a

But those links can't be accessed directly ex:link. They throw HTTP 405. Why is that?

CodePudding user response：

You can access the API by making a POST request.

Try this:

import requests
response = requests.post('https://www.cse.lk/api/marketSummery')
json.loads(response.content)

Output

{'id': 30071974,
 'tradeVolume': 5829973145.4,
 'shareVolume': 231803649,
 'tradeDate': 1660549800496}

The other APIs work too, eg. this:

url = " https://www.cse.lk/api/dailyMarketSummery"
response = requests.post(url)
json.loads(response.content)

Output

[[{'id': 12893,
   'tradeDate': 1660501800000,
   'marketTurnover': 5829973000.0,
   'marketTrades': 51086.0,
   'marketDomestic': 50376.0,
   'marketForeign': 710.0,
   'equityTurnover': 5829973000.0,
   'equityDomesticPurchase': 5709995000.0,
   'equityDomesticSales': 5592159700.0,
   'equityForeignPurchase': 119978192.0,
   'equityForeignSales': 237813568.0,
   'volumeOfTurnOverNumber': 231803648.0,
   'volumeOfTurnoverDomestic': 226246512.0,
   'volumeOfTurnoverForeign': 5320827,
   'tradesNo': 51086,
   ...

CodePudding user response：

import requests
from pprint import pp


def main(url):
    r = requests.post(url)
    pp(r.json())


main('https://www.cse.lk/api/marketSummery')

{'id': 30071974,
 'tradeVolume': 5829973145.4,
 'shareVolume': 231803649,
 'tradeDate': 1660549800496}