How to handle StaleElementReferenceException (Selenium Python)?-CodePudding

I am writing a scraper for https://etherscan.io/ using selenium in python. I am getting StaleElementReferenceException in my code when it clicks on a button and not finding any good source to solve this error.

This is the code

url = "https://etherscan.io/login?"
driver.implicitly_wait(30)
driver.get(url)
username = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_txtUserName"]')
username.send_keys('user')
password = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_txtPassword"]')
password.send_keys('pass')
login = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_btnLogin"]')
login.click()
more = driver.find_element_by_xpath('//*[@id="moreMegaMenu"]')
more.click()
word_cloud = driver.find_element_by_xpath('//*[@id="LI41"]/a').click()
soup = BeautifulSoup(driver.page_source, 'html.parser')
wait = WebDriverWait(driver, 10)
dropdown = driver.find_elements_by_css_selector('.dropdown-toggle')
for ele in dropdown:
    ele.click()
    time.sleep(1)
    account = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'.d-block')))
    driver.execute_script("arguments[0].click();", account)
    time.sleep(1)
    driver.execute_script("window.history.go(-1)")

Error Message:

StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=100.0.4896.75)

StaleElementReferenceException            Traceback (most recent call last)
<ipython-input-108-2804d338c51f> in <module>
      1 dropdown = driver.find_elements_by_css_selector('.dropdown-toggle')
      2 for ele in dropdown:
----> 3     ele.click() # this is the line where it is showing error
      4     time.sleep(1)
      5     account = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'.d-block')))

CodePudding user response：

@cruisepandey solution is ok, but I think mine is a bit simpler. Also, in my case, his solution redirects to a wrong page (to homepage).

url = "https://etherscan.io/login?"
driver.implicitly_wait(30)
driver.get(url)
username = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_txtUserName"]')
username.send_keys('user')
password = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_txtPassword"]')
password.send_keys('pass')
login = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_btnLogin"]')
login.click()
more = driver.find_element_by_xpath('//*[@id="moreMegaMenu"]')
more.click()
word_cloud = driver.find_element_by_xpath('//*[@id="LI41"]/a').click()
soup = BeautifulSoup(driver.page_source, 'html.parser')
wait = WebDriverWait(driver, 10)

num_elements = len(driver.find_elements_by_css_selector('.dropdown-toggle'))

for i in range(num_elements):
    ele = driver.find_elements_by_css_selector('.dropdown-toggle')[i]
    ele.click()
    time.sleep(1)
    account = wait.until(EC.element_to_be_clickable((By.XPATH, f'//*[@id="content"]/div[3]/div/div/div[3]/div[{i 1}]/div/div/a[1]')))
    driver.execute_script("arguments[0].click();", account)
    time.sleep(1)
    driver.execute_script("window.history.go(-1)")
    time.sleep(1)

CodePudding user response：

Problem explanation:

When you are clicking on account like below:

driver.execute_script("arguments[0].click();", account)

this is redirecting you to a new page within the same tab and windows.

However, when you go back to main page using:

driver.execute_script("window.history.go(-1)")

Now, the elements have become stale in nature.

Also, you should really click on more button instead hover over it using ActionChain.

And this line

account = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'.d-block')))

won't work for the second and the subsequent element cause .d-block is not the right locator for all other dropdown. Instead, use a relative xpath

.//following-sibling::div/a

Solution:

Keep defining the web element in the loop.

Code:

wait = WebDriverWait(driver, 20)

url = "https://etherscan.io/login?"
driver.maximize_window()
#driver.implicitly_wait(30)
driver.get(url)
username = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_txtUserName"]')
username.send_keys('user')
password = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_txtPassword"]')
password.send_keys('pass')
login = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_btnLogin"]')
login.click()

try:
    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#btnCookie"))).click()
    print('clicked on cookies button successfully.')
except:
    print('could not clicked on cookies button.')
    pass

more = wait.until(EC.element_to_be_clickable((By.XPATH, "//*[@id='moreMegaMenu']")))
ActionChains(driver).move_to_element(more).pause(2).perform()

word_cloud = wait.until(EC.element_to_be_clickable((By.XPATH, "//*[@id='LI41']/a")))
word_cloud.click()

soup = BeautifulSoup(driver.page_source, 'html.parser')

dropdown = driver.find_elements_by_css_selector('.dropdown-toggle')
i = 1
for ele in dropdown:
    ele = wait.until(EC.presence_of_element_located((By.XPATH, f"(//button[contains(@class,'dropdown-toggle')])[{i}]")))
    driver.execute_script("arguments[0].scrollIntoView(true);", ele)
    ele.click()
    time.sleep(1)
    account = wait.until(EC.element_to_be_clickable((By.XPATH, ".//following-sibling::div/a")))
    driver.execute_script("arguments[0].click();", account)
    time.sleep(1)
    driver.execute_script("window.history.go(-1)")
    i = i   1

Output:

clicked on cookies button successfully.
ETHER PRICE $3,244.85 @ 0.07608 BTC ( 1.09%)
ETHER PRICE $3,244.85 @ 0.07608 BTC ( 1.09%)
ETHER PRICE $3,244.85 @ 0.07608 BTC ( 1.09%)
ETHER PRICE $3,244.85 @ 0.07608 BTC ( 1.09%)
ETHER PRICE $3,244.85 @ 0.07608 BTC ( 1.09%)
ETHER PRICE $3,244.85 @ 0.07608 BTC ( 1.09%)
ETHER PRICE $3,244.85 @ 0.07608 BTC ( 1.09%)
ETHER PRICE $3,244.85 @ 0.07608 BTC ( 1.09%)
ETHER PRICE $3,244.71 @ 0.07609 BTC ( 1.08%)
ETHER PRICE $3,244.71 @ 0.07609 BTC ( 1.08%)
ETHER PRICE $3,244.71 @ 0.07609 BTC ( 1.08%)
ETHER PRICE $3,244.71 @ 0.07609 BTC ( 1.08%)
ETHER PRICE $3,244.71 @ 0.07609 BTC ( 1.08%)
ETHER PRICE $3,244.71 @ 0.07609 BTC ( 1.08%)
ETHER PRICE $3,244.71 @ 0.07609 BTC ( 1.08%)
ETHER PRICE $3,244.71 @ 0.07609 BTC ( 1.08%)
ETHER PRICE $3,244.71 @ 0.07609 BTC ( 1.08%)
ETHER PRICE $3,244.71 @ 0.07609 BTC ( 1.08%)
ETHER PRICE $3,244.71 @ 0.07609 BTC ( 1.08%)
ETHER PRICE $3,244.71 @ 0.07609 BTC ( 1.08%)
ETHER PRICE $3,244.53 @ 0.07608 BTC ( 1.08%)
ETHER PRICE $3,244.53 @ 0.07608 BTC ( 1.08%)
ETHER PRICE $3,244.53 @ 0.07608 BTC ( 1.08%)
ETHER PRICE $3,244.53 @ 0.07608 BTC ( 1.08%)
ETHER PRICE $3,244.53 @ 0.07608 BTC ( 1.08%)
ETHER PRICE $3,244.53 @ 0.07608 BTC ( 1.08%)
ETHER PRICE $3,244.53 @ 0.07608 BTC ( 1.08%)
ETHER PRICE $3,244.53 @ 0.07608 BTC ( 1.08%)
ETHER PRICE $3,244.53 @ 0.07608 BTC ( 1.08%)
ETHER PRICE $3,244.53 @ 0.07608 BTC ( 1.08%)
ETHER PRICE $3,244.53 @ 0.07608 BTC ( 1.08%)
ETHER PRICE $3,244.53 @ 0.07608 BTC ( 1.08%)
ETHER PRICE $3,244.53 @ 0.07608 BTC ( 1.08%)
ETHER PRICE $3,244.39 @ 0.07608 BTC ( 1.07%)
ETHER PRICE $3,244.39 @ 0.07608 BTC ( 1.07%)
ETHER PRICE $3,244.39 @ 0.07608 BTC ( 1.07%)
ETHER PRICE $3,244.39 @ 0.07608 BTC ( 1.07%)
ETHER PRICE $3,244.39 @ 0.07608 BTC ( 1.07%)
ETHER PRICE $3,244.39 @ 0.07608 BTC ( 1.07%)
ETHER PRICE $3,244.39 @ 0.07608 BTC ( 1.07%)
ETHER PRICE $3,244.39 @ 0.07608 BTC ( 1.07%)
ETHER PRICE $3,244.39 @ 0.07608 BTC ( 1.07%)
ETHER PRICE $3,244.39 @ 0.07608 BTC ( 1.07%)
ETHER PRICE $3,244.39 @ 0.07608 BTC ( 1.07%)
ETHER PRICE $3,244.39 @ 0.07608 BTC ( 1.07%)
ETHER PRICE $3,244.39 @ 0.07608 BTC ( 1.07%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,243.26 @ 0.07607 BTC ( 1.06%)
ETHER PRICE $3,242.79 @ 0.07606 BTC ( 1.04%)
ETHER PRICE $3,242.79 @ 0.07606 BTC ( 1.04%)
ETHER PRICE $3,242.79 @ 0.07606 BTC ( 1.04%)
ETHER PRICE $3,242.79 @ 0.07606 BTC ( 1.04%)
ETHER PRICE $3,242.79 @ 0.07606 BTC ( 1.04%)
ETHER PRICE $3,242.79 @ 0.07606 BTC ( 1.04%)
ETHER PRICE $3,242.79 @ 0.07606 BTC ( 1.04%)
ETHER PRICE $3,242.79 @ 0.07606 BTC ( 1.04%)
ETHER PRICE $3,242.79 @ 0.07606 BTC ( 1.04%)
ETHER PRICE $3,242.79 @ 0.07606 BTC ( 1.04%)
ETHER PRICE $3,242.79 @ 0.07606 BTC ( 1.04%)
ETHER PRICE $3,242.50 @ 0.07606 BTC ( 1.03%)
ETHER PRICE $3,242.50 @ 0.07606 BTC ( 1.03%)
ETHER PRICE $3,242.50 @ 0.07606 BTC ( 1.03%)
ETHER PRICE $3,242.50 @ 0.07606 BTC ( 1.03%)
ETHER PRICE $3,242.50 @ 0.07606 BTC ( 1.03%)
ETHER PRICE $3,242.50 @ 0.07606 BTC ( 1.03%)
ETHER PRICE $3,242.50 @ 0.07606 BTC ( 1.03%)
ETHER PRICE $3,242.50 @ 0.07606 BTC ( 1.03%)
ETHER PRICE $3,242.50 @ 0.07606 BTC ( 1.03%)
ETHER PRICE $3,242.50 @ 0.07606 BTC ( 1.03%)
ETHER PRICE $3,242.50 @ 0.07606 BTC ( 1.03%)
ETHER PRICE $3,242.14 @ 0.07607 BTC ( 1.02%)
ETHER PRICE $3,242.14 @ 0.07607 BTC ( 1.02%)
ETHER PRICE $3,242.14 @ 0.07607 BTC ( 1.02%)
ETHER PRICE $3,242.14 @ 0.07607 BTC ( 1.02%)
ETHER PRICE $3,242.14 @ 0.07607 BTC ( 1.02%)
ETHER PRICE $3,242.14 @ 0.07607 BTC ( 1.02%)
ETHER PRICE $3,242.14 @ 0.07607 BTC ( 1.02%)
ETHER PRICE $3,242.14 @ 0.07607 BTC ( 1.02%)
ETHER PRICE $3,242.14 @ 0.07607 BTC ( 1.02%)
ETHER PRICE $3,242.14 @ 0.07607 BTC ( 1.02%)
ETHER PRICE $3,242.14 @ 0.07607 BTC ( 1.02%)
ETHER PRICE $3,242.14 @ 0.07607 BTC ( 1.02%)
ETHER PRICE $3,242.14 @ 0.07607 BTC ( 1.02%)
ETHER PRICE $3,241.93 @ 0.07607 BTC ( 1.01%)
ETHER PRICE $3,241.93 @ 0.07607 BTC ( 1.01%)
ETHER PRICE $3,241.93 @ 0.07607 BTC ( 1.01%)
ETHER PRICE $3,241.93 @ 0.07607 BTC ( 1.01%)
ETHER PRICE $3,241.93 @ 0.07607 BTC ( 1.01%)
ETHER PRICE $3,241.93 @ 0.07607 BTC ( 1.01%)
ETHER PRICE $3,241.93 @ 0.07607 BTC ( 1.01%)
ETHER PRICE $3,241.93 @ 0.07607 BTC ( 1.01%)

and so on..