TimeoutException when accessing element in modal using Selenium-CodePudding

Problem

I want to extract the version histories of apps in Apple App Store's website. They're elements within a modal container, which pops up upon clicking the Version History button.

I can't for the life of me figure out why TimeoutException occurs intermittently when I try to access the elements in the modal - whether I've written the CSS_SELECTOR wrong or haven't extended the timeout long enough. On occasion, the code's successful and returns me my desired list of version histories, but most of the time, it doesn't work.

I'll appreciate any help.

Relevant HTML

Website in question: Twitter App Store.

Button to click:

<div class="version-history">
    <button class="we-modal__show link section__nav__see-all-link" id="modal-trigger-ember215599187" data-metrics-click="{&quot;actionType&quot;:&quot;open&quot;,&quot;targetType&quot;:&quot;button&quot;,&quot;targetId&quot;:&quot;ModalVersionHistory&quot;}" type="button">Version History</button>
</div>

Modal when button's not clicked:

<div id="modal-container"></div>

Modal when button's clicked:

<div id="modal-container">
<div class="we-modal we-modal--page-overlay we-modal--open version-history" role="dialog" aria-hidden="false" aria-labelledby="modal-trigger-ember148" tabindex="0">
   <div class="we-modal__content large-10 medium-12 " id="modal-content-ember148" role="document" tabindex="0" data-focus-method="mouse">
      <div class="we-modal__content__wrapper">
            
        <h3 class="version-history__headline">Version History</h3>
        <ul class="version-history__items">
            <li class="version-history__item">
               <h4 class="version-history__item__version-number">8.86</h4>
               <time data-test-we-datetime="" datetime="2021-10-07T00:00:00.000Z" aria-label="October 7, 2021" class="version-history__item__release-date">Oct 7, 2021</time>
               <div class="we-truncate we-truncate--multi-line we-truncate--interactive  version-history__item__release-notes" dir="" aria-label="">

                  <div data-clamp="" class="we-clamp">
                      <p dir="ltr" data-test-bidi="">We made improvements and squashed bugs so Twitter is even better for you.</p>
                  </div>

               </div>
            </li>

            <!-- More li content follows -->

       </div>
    </div>
  </div>
</div>

My code

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless")

url = "https://apps.apple.com/us/app/twitter/id333903271?ign-mpt=uo=4"
driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)
driver.get(url)

# click on Version History button
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, '.version-history > button[class="we-modal__show link section__nav__see-all-link"]')))
button = driver.find_element_by_css_selector('.version-history > button[class="we-modal__show link section__nav__see-all-link"]')
button.click()

# get all versions into an array
modal = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'div[id="modal-container"]')))
version_items = modal.find_elements_by_tag_name("h4")
versions = list(map(lambda v: v.text, version_items))

driver.close()

The error I experience

TimeoutException                          Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_60716/874757843.py in <module>
     18     #version_items = driver.find_elements_by_css_selector('#modal-container > h4')
     19 
---> 20     modal = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'div[id="modal-container"]')))
     21     version_items = modal.find_elements_by_tag_name("h4")
     22 

~\miniconda3\lib\site-packages\selenium\webdriver\support\wait.py in until(self, method, message)
     78             if time.time() > end_time:
     79                 break
---> 80         raise TimeoutException(message, screen, stacktrace)
     81 
     82     def until_not(self, method, message=''):

TimeoutException: Message:

CodePudding user response：

Change the line where you get modal from visibility_of_element_located to presence_of_element_located, also; I recommend using XPATH.

from selenium import webdriver

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
import time

url = "https://apps.apple.com/us/app/twitter/id333903271?ign-mpt=uo=4"
driver = webdriver.Chrome('chromedriver.exe')
driver.get(url)

# click on Version History button
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '/html/body/div[4]/div/main/div[2]/section[4]/div[1]/div/button')))
button = driver.find_element_by_xpath('/html/body/div[4]/div/main/div[2]/section[4]/div[1]/div/button')
button.click()

# get all versions into an array
modal = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '/html/body/div[5]')))

version_items = modal.find_elements_by_tag_name("h4")
versions = list(map(lambda v: v.text, version_items))
print(versions)
time.sleep(10)