I'm scraping a website for dividend information, and the selenium query is taking a very long time because it gets stuck on driver.get(). I tried something like this:

URL = 'https://www.nasdaq.com/market-activity/stocks/aapl/dividend-history'

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import selenium as se

options = se.webdriver.FirefoxOptions()
driver = se.webdriver.Firefox()
driver.get(URL)
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located(
        (By.XPATH, '//div')))
driver.execute_script("window.stop();")

but that doesn't work because WebDriverWait(driver, 10) doesn't run until after driver.get(URL), so it just hangs on driver.get(URL). I also tried:

from selenium.common.exceptions import TimeoutException
driver.set_page_load_timeout(10)
try:
    driver.get(URL)
except TimeoutException:
    driver.execute_script("window.stop();")

This at least limits the query to 10 seconds, because the element I need is always loaded in the first 10 seconds, but I'd rather not have to wait 10 seconds. Is there a way to interrupt the driver.get(URL) when the appropriate element is loaded on the page?

CodePudding user response：

There is a less complex way of obtaining that data, without the overheads of Selenium setup. The data in that dividends table is loaded dynamically, from an api (you can see it in Dev tools - Network tab). The following code will get you that data:

import requests
import pandas as pd

headers= {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36'}

url = 'https://api.nasdaq.com/api/quote/AAPL/dividends?assetclass=stocks'
r = requests.get(url, headers=headers)
df = pd.DataFrame(r.json()['data']['dividends']['rows'])
print(df)

This prints out in the terminal:

​