Exiting a function that uses Selenium - how to handle an error-CodePudding

I have a mining script where I use Selenium to go trough the results pages. My function is the following:

def moving_pages():
    global driver
    driver.get(url)


    while driver.find_element_by_class_name('next-page-btn'):
        try:
            button = driver.find_element_by_class_name('next-page-btn')
            time.sleep(4)
            button.click()
            getdata(driver.current_url)
            new_url = (driver.current_url)
            print(new_url)
        except NoSuchElementException:
            print("NO MORE NEXT BUTTON")

The function works well, it moves trough the pages, however on the last page, there is no 'next-page-btn', therefore I need to handle this. Unfortunattly when i run my code I still get an error, the error being:

Traceback (most recent call last):
  File "/Users/tdonov/Desktop/Python/Realestate Scraper/Selenium_updated_mining_script copy.py", line 188, in <module>
    moving_pages()
  File "/Users/tdonov/Desktop/Python/Realestate Scraper/Selenium_updated_mining_script copy.py", line 160, in moving_pages
    while driver.find_element_by_class_name('next-page-btn'):
  File "/Users/tdonov/Desktop/Python/venv/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 754, in find_element_by_class_name
    return self.find_element(by=By.CLASS_NAME, value=name)
  File "/Users/tdonov/Desktop/Python/venv/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 1238, in find_element
    return self.execute(Command.FIND_ELEMENT, {
  File "/Users/tdonov/Desktop/Python/venv/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 418, in execute
    self.error_handler.check_response(response)
  File "/Users/tdonov/Desktop/Python/venv/lib/python3.10/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".next-page-btn"}
  (Session info: chrome=96.0.4664.55)
Stacktrace:
0   chromedriver                        0x0000000100e61519 __gxx_personality_v0   577193
1   chromedriver                        0x0000000100ded663 __gxx_personality_v0   102387
2   chromedriver                        0x00000001009b4d28 chromedriver   171304
3   chromedriver                        0x00000001009e9c9c chromedriver   388252
4   chromedriver                        0x0000000100a1bb04 chromedriver   592644
5   chromedriver                        0x0000000100a06ddd chromedriver   507357
6   chromedriver                        0x0000000100a198ab chromedriver   583851
7   chromedriver                        0x0000000100a070a3 chromedriver   508067
8   chromedriver                        0x00000001009dd5ae chromedriver   337326
9   chromedriver                        0x00000001009de8d5 chromedriver   342229
10  chromedriver                        0x0000000100e1d75f __gxx_personality_v0   299247
11  chromedriver                        0x0000000100e3469a __gxx_personality_v0   393258
12  chromedriver                        0x0000000100e39f5b __gxx_personality_v0   415979
13  chromedriver                        0x0000000100e35a3a __gxx_personality_v0   398282
14  chromedriver                        0x0000000100e1237c __gxx_personality_v0   253196
15  chromedriver                        0x0000000100e51458 __gxx_personality_v0   511464
16  chromedriver                        0x0000000100e515e1 __gxx_personality_v0   511857
17  chromedriver                        0x0000000100e68968 __gxx_personality_v0   606968
18  libsystem_pthread.dylib             0x00007fff71cb6109 _pthread_start   148
19  libsystem_pthread.dylib             0x00007fff71cb1b8b thread_start   15


Process finished with exit code 1

My question is, how do I modify the function in order to print as it should do now NO MORE NEXT BUTTON and exit the function so it can continue on?

CodePudding user response：

while driver.find_element_by_class_name('next-page-btn'): is the offending line. You're catching one of the exceptions that it may raise inside the loop's scope, but not in the loop condition.

Since you seem to want to break out of the loop once there are no more next-page-btns, you could do something like:

def moving_pages():
    global driver  # I recommend you pass this as a parameter instead
    driver.get(url)


    while True:
        try:
            button = driver.find_element_by_class_name('next-page-btn')
        except NoSuchElementException:
            print("NO MORE NEXT BUTTON")
            break
        time.sleep(4)
        button.click()
        getdata(driver.current_url)
        new_url = (driver.current_url)
        print(new_url)

An alternative would be to wrap that loop itself in a try-except, but it's typically good to try to reduce nesting and "range" of a the try-except blocks (note how I narrowed down the try-except to only wrap the line that may raise the exception you're catching).