Home > Software engineering >  selenium python TimeOutException during automation scraping
selenium python TimeOutException during automation scraping

Time:12-27

is TimeOutException avoidable or is there any some way to fix this error? the problem is, its randomly occur in any lines, so i cant specific which lines got TimeOutException as for the code below, it appears on detail_topic. the error frequently happend when using actions such as click

#loop how many pages
for i in range(0, 10):
    #loop how many topic
    total = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, '//*[@id="topic-list"]/card-topic')))
    #loop every topic
    for i in range(1, len(total)):
        
        time.sleep(0.5)
        topic = driver.find_element(By.XPATH, '//*[@id="topic-list"]/card-topic[{}]'.format(i))
        time.sleep(0.5)
        shadow1 = driver.execute_script("return arguments[0].shadowRoot", topic)
        time.sleep(0.5)
        shadow1.find_element(By.CSS_SELECTOR, 'a').send_keys(u'\ue007')
    
        detail_topic = WebDriverWait(driver, 100).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'detail-topic')))
        shadow2 = driver.execute_script("return arguments[0].shadowRoot", detail_topic)
        title = shadow2.find_element(By.CSS_SELECTOR, '.h2').text
        title_data3ab.append(title)

        time.sleep(0.5)
        driver.back()
    
    pagination = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, f'//paginate-button')))
    shadow_pagination = driver.execute_script("return arguments[0].shadowRoot", pagination)
    next_button = shadow_pagination.find_element(By.CSS_SELECTOR, 'a.page-next')
    actions = ActionChains(driver)
    time.sleep(0.5)
    actions.move_to_element(next_button)
    time.sleep(0.5)
    actions.click(next_button).perform()

i havent try&except since the error occur in random lines, so i need to figure out which one is the main issue, any help would be appreciated thank you

CodePudding user response:

anyway somehow i fix the issue by adding adblock on my webdriver chrome_options.add_extension('yourextensionchoice.crx') #had to be in the same path

  • Related