Unable to save json data using python and selenium-CodePudding

I am trying to save json data from https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY. The data.json file is empty. Here is the code

import time, json

from selenium import webdriver
 
driver = webdriver.Firefox()
 
url = "https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY"
 
driver.get(url)
 
time.sleep(5)

button = driver.find_element_by_id("rawdata-tab")
 
button.click()


data = driver.find_element_by_class_name("data").text

d = json.loads(data)

with open('data.json', 'w') as f:
    json.dumps(d, default=lambda o: '<not serializable>')


time.sleep(10)

driver.close()

Can't find where I am going wrong, is there a better way to achieve this?

CodePudding user response：

As, I have mentioned in my comment, selenium is used for web scraping or more generally mimicking human actions on webpages. But your URL doesn't results in a webpage. Now, from what I understand, try this instead:

import json, requests

url = "https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY"
 
response = requests.get(url)

d = response.json()

with open('data.json', 'w') as f:
    json.dumps(d, default=lambda o: '<not serializable>')

As your url already results in json, here I am just using requests to get it and it results in a valid json which you can then write however you want to a file.

PS: Try to understand whatever you're writing in code as this will help you debug, otherwise you are always going to get stuck somewhere.

CodePudding user response：

just add a user-agent to your request headers

import requests
headers={'User-Agent':'Mozilla/5.0 (Android 4.2.1; Mobile; rv:32.0) Gecko/32.0 Firefox/32.0'}
url = "https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY"
data = requests.get(url, headers=headers)
with open('data.json', 'w') as f:
    f.write(data.text)