Error when extracting a JavaScript value from a variable with Selenium python-CodePudding

I'm attempting to get the extract the value of the variable remainingTimeString on this website in python using Selenium webdriver. I'm trying to use the driver.execute_script() function. Here is my code:

import selenium.webdriver

options = selenium.webdriver.FirefoxOptions()
options.add_argument("--headless")

driver = selenium.webdriver.Firefox(options=options)

driver.get('https://shopgoodwill.com/item/151632327')
print(driver.execute_script("return remainingTimeString"))

However, when I run it, I get:

selenium.common.exceptions.JavascriptException: Message: ReferenceError: remainingTimeString is not defined

What should I do? the variable is clearly in a script when I check the HTML source. Thanks!

CodePudding user response：

That data is being pulled dynamically from an API, after page loads, so your options are - either use a WebDriverWait for that element (or an implicit wait for that matter), or use a less complex solution, like below (without selenium), where you inspect the Network Tab in Dev Tools, locate the API where data is being pulled from, and scrape that API directly:

import requests
import pandas as pd


headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
}

url = 'https://buyerapi.shopgoodwill.com/api/ItemDetail/GetItemDetailModelByItemId/151632327'

r = requests.get(url, headers=headers)
df = pd.json_normalize(r.json())
print(df['remainingTime'][0])

Result printed in terminal:

'17h 56m '

The code above is extracting remainingTime only from that dataframe. There are detailed product infos in that json, so you can get other data as well, if you need. Python requests documentation can be found at https://requests.readthedocs.io/en/latest/

Also pandas documentation: https://pandas.pydata.org/pandas-docs/stable/index.html