Handling website errors with selenium python-CodePudding

I am scraping a website with selenium and send an alert, if something specific happens. Generally, my code works fine, but sometimes the website doesn't load the elements or the website has an error message like: "Sorry, something went wrong! Please refresh the page and try again!" Both times, my script waits until elements are loaded, but they don't and then my program doesn't do anything. I usually use requests and Beautifulsoup for web scraping, so I am not that familiar with selenium and I am not sure how to handle these errors, because my code doesn't send an error message and just waits, until the elements load, which will likely never happen. If I manually refresh the page, the program continues to work. My idea would be something like: If it takes more than 10 seconds to load, refresh the page and try again. My code looks somewhat like this:

def get_data():
    data_list = []
    while len(data_list) < 3:
        try:
            data = driver.find_elements_by_class_name('text-color-main-secondary.text-sm.font-bold.text-left')
            count = len(data)
            data_list.append(data)
            driver.implicitly_wait(2)
            time.sleep(.05)
            driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
            WebDriverWait(driver, 3).until(EC.visibility_of_element_located((By.CLASS_NAME,
                                                                         'text-color-main-secondary.text-sm.font-bold.text-left'.format(
                                                                             str(
                                                                                 count   1)))))
        except TimeoutException:
            break

    text = []
    elements = []
    for i in range(len(data_list)):
        for j in range(len(data_list[i])):
            t = data_list[i][j].text
            elements.append(data_list[i][j])
            for word in t.split():
                if '#' in word:
                    text.append(word)
    return text, elements


option = webdriver.ChromeOptions()
option.add_extension('')
path = ''
driver = webdriver.Chrome(executable_path=path, options=option)
driver.get('')
login(passphrase)
driver.switch_to.window(driver.window_handles[0])
while True:
    try:
        infos, elements = get_data()
        data, message = check_data(infos, elements)
        if data:
            send_alert(message)
        time.sleep(600)
        driver.refresh()
    except Exception as e:
        exception_type, exception_object, exception_traceback = sys.exc_info()
        line_number = exception_traceback.tb_lineno
        print("an exception occured - {}".format(e)   " in line: "   str(line_number))

CodePudding user response：

You can use try and except to overcome this problem. First, let's locate the element with a 10s waiting time if the element is not presented you can refresh the page. here is the basic version of the code

try:
  # wait for 10s to load element if it did not load then it will redirect to except block
  WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CLASS_NAME,'text-color-main-secondary.text-sm.font-bold.text-left'.format(str(count   1)))))
except:
  driver.refresh()
  # locate the elemnt here again