Selenium - couldn't click next page-CodePudding

I'm trying to click the next button (>) and then repeat until the last page is reached. I've tried searching for solutions from similar questions but I couldn't figure out what's wrong/make it work.

Here is my code, thank you!

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as Wait
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException

driver = webdriver.Chrome("C:/Users/krish/Desktop/chromedriver_win32/chromedriver.exe")

driver.get('https://www.cnbcindonesia.com/tag/pasar-modal')
wait = WebDriverWait(driver,30)


while True:
 try:
    element = wait.until(EC.element_to_be_clickable((By.XPATH, "//a[@class = 'icon icon-angle-right']/i")))
    if (element != 0):
        element.click()
 except TimeoutException as ex:
        break

CodePudding user response：

Your xpath looks incorrect, please try the below

Xpath

//i[@class='icon icon-angle-right']

CSS Selector

.icon.icon-angle-right

There are multiple approaches to address this issue and a couple of them are as follows:

As you intent to invoke click() you need to induce WebDriverWait in conjunction with the WebDriverWaitWebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".icon.icon-angle-right"))).click()

Using XPATH:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//i[@class='icon icon-angle-right']"))).click()

Incase the error ...another element obscures it... still persists first you need to induce WebDriverWait inconjunction with the expected_conditions for the invisibility_of_element() of the blocking element as follows:

Using CSS_SELECTOR:

WebDriverWait(driver, 20).until(EC.invisibility_of_element((By.CSS_SELECTOR, ".icon.icon-angle-right")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".icon.icon-angle-right"))).click()

Using XPATH:

WebDriverWait(driver, 20).until(EC.invisibility_of_element((By.XPATH, "//i[@class='icon icon-angle-right']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//i[@class='icon icon-angle-right']"))).click()

If the issue still persists you can use the execute_script() method as follows:

Using CSS_SELECTOR:

WebDriverWait(driver, 20).until(EC.invisibility_of_element((By.CSS_SELECTOR, ".icon.icon-angle-right")))
driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".icon.icon-angle-right"))))

Using XPATH:

WebDriverWait(driver, 20).until(EC.invisibility_of_element((By.XPATH, "//i[@class='icon icon-angle-right']")))
driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//i[@class='icon icon-angle-right']"))))

Note: : You have to import below

from selenium.webdriver.support.ui import WebDriverWait       
from selenium.webdriver.common.by import By       
from selenium.webdriver.support import expected_conditions as EC

CodePudding user response：

You can do it easy with playwright

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.webkit.launch(headless=False)
    baseurl = "https://www.cnbcindonesia.com/tag/pasar-modal"
    page = browser.new_page()
    page.goto(baseurl)
    next_page = page.wait_for_selector("//i[@class='icon icon-angle-right']")
    next_page.click()
    browser.close()

Also if you want to paginate, you can do it with changing just URL param.
Go to page 87: https://www.cnbcindonesia.com/tag/pasar-modal/87?kanal=&tipe=

CodePudding user response：

Figured it out, finally works by using execute_script() method

while True:
    try:
        time.sleep(2)
        driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".icon.icon-angle-right"))))
    except NoSuchElementException:
        break

Thank you all for your answers