Python Selenium StaleElementReferenceException: Message: stale element reference: element is not att-CodePudding

I am trying to scrape data from a website using selenium and chromedriver in python. After the login to the website, I have to select some values from one dropdown menu but I am unable to click on the dropdown menu using code. I have written the code as:

from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver,60)
element=wait.until(EC.element_to_be_clickable((By.XPATH,"//input[@id='ContentPlaceHolder1_rcmbCapacityTranch_Input']")))
element.click()

When I run the above lines of code, I get StaleElementReferenceException: Message: stale element reference: element is not attached to the page document but when I include time.sleep(2) as:

time.sleep(2)
element=wait.until(EC.element_to_be_clickable((By.XPATH,"//input[@id='ContentPlaceHolder1_rcmbCapacityTranch_Input']")))
element.click()

Now, it executed successfully and dropdown menu gets clicked and values are shown but sometimes 2 seconds time break is not enough, I have to give 5 seconds and sometimes I have to give 10 seconds time break which slow down the execution and sometimes this 10 seconds time break fails also.
I have also tried following lines of code as:

staleElement = True
while(staleElement):
    try:
        xpath="//input[@id='ContentPlaceHolder1_rcmbCapacityTranch_Input']"
        element=wait.until(EC.element_to_be_clickable((By.XPATH,xpath)))
        element.click()
        staleElement = False
    except StaleElementReferenceException:
        staleElement = True

The above lines of code executed successfully but dropdown menu is not clicked and does not show the values but when I re-executed the same lines of code as above in jupyter notebook cell then dropdown menu gets selected and show the values.
So, Can anyone please help me to explain the reason behind the things which I mentioned above and give me the permanent solution to resolve the issue without sleep().
Any help would be appreciated.

CodePudding user response：

Since I have no credentials to enter that site I can only guess.
So my guess is: that site uses dynamic DOM.
It means that after the initial appearance the web elements continue re-build, so even after Selenium caught some specific element to be clickable or visible this element may instantly continue changing so that the initial element reference becomes stale.
You can try treat this problem by 2 ways:

Set first wait condition, after that set a delay and after that another wait condition. Something like this:

from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver,60)
wait.until(EC.element_to_be_clickable((By.XPATH,"//input[@id='ContentPlaceHolder1_rcmbCapacityTranch_Input']")))
time.sleep(2)
wait.until(EC.element_to_be_clickable((By.XPATH,"//input[@id='ContentPlaceHolder1_rcmbCapacityTranch_Input']"))).click()

After you caught the element try clicking it. In case Stale element exception is raised wait until the element is clickable again and try clicking it again until click is succeed. With such loop you will normally make a click during 2-3 iterations.
So please try this:

from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver,60)
succeed = False
while !succeed:
    try:
        wait.until(EC.element_to_be_clickable((By.XPATH,"//input[@id='ContentPlaceHolder1_rcmbCapacityTranch_Input']"))).click()
        succeed = True
    except:
        pass