Can't get JSON response to be printed or stored in variable selenium python-CodePudding

Using below code as an example to get what i want in res variable

from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
import time


options = Options()
options.add_argument("--disable-blink-features=AutomationControlled")
#options.headless = True

driver = webdriver.Chrome('C:/Users/Amanr/Downloads/chromedriver_win32 v106/chromedriver', options=options)

driver.get("https://www.")
time.sleep(5)
res = driver.execute_script('''window.open('https://',"_blank");''')
main_window = driver.window_handles[1]
driver.switch_to.window(main_window)
ps = driver.page_source
print(ps)

The result i am getting now is via page source but i want it as clear JSON response. it also doesn't have full data as in the link something like this:

{"identifier":"OPTIDXBANKNIFTY27-10-2022CE41200.00","name":"BANKNIFTY","grapthData":[[1666689300000,359.2],[1666689301000,386.35],[1666689302000,393.45],[1666689303000,397.05],[1666689304000,385.8],[1666689305000,383.25],[1666689306000,378.95],[1666689307000,370.45],[1666689308000,369.15],[1666689309000,372.3],[1666689310000,383.75],[1666689311000,391.65],[1666689312000,400.9],[1666689313000,393.85],[1666689314000,402.7],[1666689315000,397.2],[1666689316000,391.5],[1666689317000,391.95],[1666689318000,391.85],[1666689319000,385.4],[1666689350000,380.55],[1666689351000,380.25],[1666689352000,377.15],[1666689353000,376.1],[1666689354000,373.85],[1666689355000,370

Is there any way i can get directly get the response as json.

CodePudding user response：

The api endpoint you confirmed in comment above returns the following information - see below:

import requests
import pandas as pd

pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', None)

big_list = []
url = 'https://www.nseindia.com/api/chart-databyindex?index=OPTIDXBANKNIFTY27-10-2022CE41200.00'
headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36 Edg/106.0.1370.47'
}
s = requests.Session()
s.headers.update(headers)
s.get('https://www.nseindia.com/option-chain')

r = s.get(url)
for x in r.json()['grapthData']:
    big_list.append(set(x))
df = pd.DataFrame(big_list, columns = ['Timestamp', 'Value'])
print(df)

Result in terminal:

Timestamp   Value
0   1666689300000   359.20
1   1666689301000   386.35
2   1666689302000   393.45
3   1666689303000   397.05
4   1666689304000   385.80
... ... ...
7691    1666711732000   121.85
7692    1666711733000   121.55
7693    1666711734000   121.20
7694    1666711735000   121.15
7695    1666711736000   121.10
7696 rows × 2 columns

I randomly named the columns, that data probably means something else, nonetheless there it is.

CodePudding user response：

You can grab all the required data from API as json format as follows:

import requests

api_url = "https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY"
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36',
  
   }

req = requests.get(api_url,headers=headers).json()
print(req.get('filtered').get('data'))