Issue in navigating ASPX pagination using Selenium-CodePudding

I would like to scrape this URL to iterate over pages and get their content. That's what I have tried:

driver.get("https://www.shoroukbookstores.com/books/publishing-house.aspx?id=429c2704-5fa3-43b0-9ad7-c2ec9062beb3")
page_count = 1
while True:
    page_count  = 1
    try:
        WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='Body_AspNetPager']/a[%s]" % page_count))).click()                                                          
        print("Next page clicked")

    except NoSuchElementException:
        print("No more pages")
        break

driver.quit()

and I get 10 pages out of 102; specifically the visible page numbers in the URL. How to paginate all pages in this URL?

CodePudding user response：

Instead of clicking on every number, use the next button.

driver.implicitly_wait(5)
driver.get("https://www.shoroukbookstores.com/books/publishing-house.aspx?id=429c2704-5fa3-43b0-9ad7-c2ec9062beb3")

while True:

    try:
        navigation_links = driver.find_element_by_xpath(f"//*[@id='Body_AspNetPager']").find_elements_by_tag_name("a")
        next_page = len(navigation_links) - 1 # next page button is second from behind
        driver.find_element_by_xpath(f"//*[@id='Body_AspNetPager']/a[{next_page}]").click()
        print("Next page clicked")

    except NoSuchElementException:
        print("No more pages")
        break

driver.quit()