How to break a loop if certain element is disabled and get text from multiple pages in Selenium Pyth-CodePudding

I am a new learner for python and selenium. I have written a code to extract data from multiple pages but there is certain problem in the code.

I am not able to break the a while loop function which clicks on next page until there is an option. The next page element disables after reaching the last page but code sill runs.

xpath: '//button[@aria-label="Next page"]'

Full SPAN:

I am able to get the list of data which I want to extract from the webpage but I am getting on the last page data when I close the webpage from my end, ending the while loop.

Full Code:

opts = webdriver.ChromeOptions()
opts.headless = True
driver = webdriver.Chrome(ChromeDriverManager().install())
base_url = "XYZ"
driver.maximize_window()
driver.get(base_url)
driver.set_page_load_timeout(50)
element = WebDriverWait(driver, 50).until(EC.presence_of_element_located((By.ID, 'all-my-groups')))
driver.find_element(by=By.XPATH, value = '//*[@id="sim-issueListContent"]/div[1]/div/div/div[2]/div[1]/span/div/input').send_keys('No Stock')

dfs = []
page_counter = 0
while True:

    wait = WebDriverWait(driver, 30)
    wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@class, 'alias-wrapper sim-ellipsis sim-list--shortId')]")))
    cards = driver.find_elements_by_xpath("//div[contains(@class, 'alias-wrapper sim-ellipsis sim-list--shortId')]")
    sims = []
    for card in cards:
        sims.append([card.text])

    df = pd.DataFrame(sims)    
    dfs.append(df)
    
    print(page_counter)
    page_counter =1
    try:
        wait.until(EC.element_to_be_clickable((By.XPATH,'//button[@aria-label="Next page"]'))).click()
    
    except:
        break

driver.close()
driver.quit()

I am also, attaching the image of the class and sorry I cannot share the URL as it of private domain.

CodePudding user response：

The easiest option is to let your wait.until() fail via timeout when the "Next page" button is missing. Right now your line wait = WebDriverWait(driver, 30) is setting the timeout to 30 seconds; assuming the page normally loads much faster than that, you could change the timeout to be 5 seconds and then the loop will end faster once you're at the last page. If your page load times are sometimes slow then you should make sure the timeout won't accidentally cut off too early; if the load times are consistently fast then you might be able to get away with an even shorter timeout interval.

Alternatively, you could look through the specific target webpage more carefully to find some element that a) is always present and b) can be used to determine whether we're on the final page or not. Then you could read the value of that element and decide whether to break the loop before trying to find the "Next page" button. This could save a couple of seconds of waiting on the final loop iteration (avoid waiting for timeout) but may not be worth the trouble.

CodePudding user response：

Change the below condtion

try:
        wait.until(EC.element_to_be_clickable((By.XPATH,'//button[@aria-label="Next page"]'))).click()
    
    except:
        break

as shown in the below pseduocode @disabled is the diff that will make sure to exit the while loop if the button is disabled.

if(driver.find_elements_by_xpath('//button[@aria-label="Next page"][@disabled]'))).size()>0)
    break
else
    driver.find_element_by_xpath('//button[@aria-label="Next page"]').click()