Python & Selenium: How to avoid StaleElementReferenceException error from dividend.com-CodePudding

I'd like to get company names from https://www.dividend.com/dividend-champions/ with Python & Selenium.

But the following error is displayed after some company names are printed.

StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

How can I avoid this error?

Curiously enough, some company names are printed before occurrence of this error.

It would be appreciated if you could give me some hint.

Python

driver.get('https://www.dividend.com/dividend-champions/')
companies = driver.find_elements(by=By.CLASS_NAME, value="m-table-long-name")
print(len(companies)) #30
for company in companies:
  print(company.text)

CodePudding user response：

StaleElementReferenceException means selected element may no longer in html dom.However,there are plenty of reasons behind this exception like incorrect element selection,load time and so on. So You need to use webdriverwait.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

from selenium.webdriver.chrome.options import Options

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC


option = webdriver.ChromeOptions()
option.add_argument("start-maximized")

#chrome to stay open
option.add_experimental_option("detach", True)

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=option)
driver.get('https://www.dividend.com/dividend-champions/')


names = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, '//*[@]/div[@]')))

for name in names:
    company = name.find_element(By.XPATH,'.//a[@]').text
    print(company)

Output:

Eversource Energy
Andersons Inc.
J.M. Smucker Co.
Cambridge Bancorp
Factset Research Systems Inc.
Graco Inc.
First Of Long Island Corp.
Stepan Co.
MDU Resources Group Inc
W. P. Carey Inc
Church & Dwight Co., Inc.
Lincoln Electric Holdings, Inc.
Matthews International Corp. - Ordinary Shares - Class A
New Jersey Resources Corporation
Bank OZK
Caterpillar Inc.
Brown & Brown, Inc.
Carrier Global Corp
Polaris Inc
International Business Machines Corp.
NextEra Energy Inc
Realty Income Corp.
RenaissanceRe Holdings Ltd
Cullen Frost Bankers Inc.
Otis Worldwide Corp
Roper Technologies Inc
West Pharmaceutical Services, Inc.
Bancfirst Corp.
Albemarle Corp.
Aptargroup Inc.

WebdriverManager