Having issues grabbing the xhr/json data of a site-CodePudding

The data for his table is behind ajax request. I'm still learning but I don't want to resort to using selenium yet as looking at the table element it's going to be hard to arrange the data into a data frame. How can I get this and what sort of things should I look for in the future with sites like these.

I have tried using all the request headers but I still want able to get the JSON data.

https://www.barchart.com/futures/quotes/ESU22/options/MI7N22

CodePudding user response：

You need one header x-xsrf-token and other cookie laravel_token to request API.
How can you get this cookies? Only when you open real browser (selenium or playwright) and get them after you can send simple request to API.
I got this cookies from my browser copied API cURL.
Here you have some example code:

import requests

headers = {
    'authority': 'www.barchart.com',
    'accept': 'application/json',
    'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8,es;q=0.7,ru;q=0.6',
    'cookie': 'laravel_token=eyJpdiI6IkkxM1I5UEZ5MStweUs3TllON0o4THc9PSIsInZhbHVlIjoiUWYyaUJUR1RlanFoTmhvdFVXb1NNM3MwTmtmOFZHcVpxSmpHNkNZc0htVEdQNEdzY0NyeHlkTmdJVUNsY3hqNithTi9pRjZLeHVzQVU3YnVkZHpmT3hUaUlvbVdzZnhVNzFZdzVTVFVEQzZ6UVBJNjIwUzN3Yk9rbVhOUXNSMm5JOUluK3ZzRTJjV3VKSTZ0SDVZN21vMnVQZWdRNjhLcUgwck9qMlF5RVVWaG9JbVUzVEdRb1p4ZXVVK0MydFhGdjlYQUhmUFhBb3JnWkt5SHBVdzRjMHlteVphVTZuTXlvUHlsOUljdjNvUzVtVHdkOUdVNjZCNTJWVDJsaFptWWlTbWlzOXFXM2pXNGZrWktyejlCOXR3QlgxVkdwUHhGWXVwVjdqRnVPNTZKOTR3OGZqS3JPQTNxdVhEb1JCUzEiLCJtYWMiOiJmOGUzMDk4MDVjYzdkNmJiYTM0MTRiZmE5Mjc1NDUxZGM3NTY1NWFlMWRiMzdmMGViMTA5MTY0NmFiMGQwNTFlIn0=; ',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36',
    'x-xsrf-token': 'eyJpdiI6InNCL1BqSTFKN3pkdGcyL0VDMTNTbmc9PSIsInZhbHVlIjoiQ0FKQmdzcVRqSGkzTXJ6c0JKdlNkdzR3TzU4RWxQZVBRSC9kVDdxcVFQWTMwUnlrakdyWEwyd01tbUgzOGpDdEt4UHRraU05WHBwOHNFclQ5RXFyck1CTktiOUNLTUtzUEpkdkR2V3pBMm1LbTlVSlpOUFZjUlhXL1ZIbGVFMFgiLCJtYWMiOiJhMTgyNDQ4NjJhYTViOGU5YmE5ODc1MzlmNTcwYTBkYTExOThlNGIwM2I3OWU3MGNhZDQ5ZTNmODNhMGNiZTU2In0=',
}
params = {
    'symbol': 'MI7N22',
    'list': 'futures.options',
    'fields': 'strike,highPrice,lowPrice,lastPrice,priceChange,bidPrice,askPrice,volume,openInterest,premium,tradeTime,longSymbol,symbolCode,symbolType,hasOptions',
    'meta': 'field.shortName,field.description,field.shortName,field.type,lists.lastUpdate',
    'orderBy': 'strike',
    'orderDir': 'asc',
    'hasOptions': 'true',
    'raw': '1',
}

response = requests.get('https://www.barchart.com/proxies/core-api/v1/quotes/get', params=params, headers=headers)
json = response.json()
print(json["data"][0])

OUTPUT:

{'strike': '1,900.00C', 'highPrice': '1,774.00', 'lowPrice': '1,774.00', 'lastPrice': '1,774.00', 'priceChange': ' 4.25', 'bidPrice': 'N/A', 'askPrice': 'N/A', 'volume': 'N/A', 'openInterest': 'N/A', 'premium': '88,700.00', 'tradeTime': '06/17/22', 'longSymbol': 'MI7N2|1900C', 'symbolCode': 'FUTOPT', 'symbolType': 12, 'hasOptions': 'No', 'raw': {'strike': 1900, 'highPrice': 1774, 'lowPrice': 1774, 'lastPrice': 1774, 'priceChange': 4.25, 'bidPrice': 0, 'askPrice': 0, 'volume': None, 'openInterest': None, 'premium': 88700, 'tradeTime': 1655501850, 'longSymbol': 'MI7N2|1900C', 'symbolCode': 'FUTOPT', 'symbolType': 12, 'hasOptions': False}}

Maybe you will need to change the value of this cookie and header. (i don't know how long are they valid and if they are linked to my IP address).
I hope I have been able to help you.