Selenium driver get_log() stops suddenly-CodePudding

I have a script that creates multiple selenium.webdriver-instances, executes a js-script and reads the resulting logs of them. Most of the time the script runs without problems, but in a few cases the logs suddenly stop after running for a while. I am not sure how to mimic the error.

My headless webdriver is based on this answer and defined as follows:

import threading
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

threadLocal = threading.local()
DRIVER_PATH = '/chromedriver'

def create_driver_headless() -> webdriver.Chrome:
    driver = getattr(threadLocal, 'driver', None)
    if driver is None:
        dc = DesiredCapabilities.CHROME
        dc['goog:loggingPrefs'] = {'browser': 'ALL'}
        chrome_options = Options()
        chrome_options.add_argument('--disable-infobars')
        chrome_options.add_argument('--disable-gpu')
        chrome_options.add_argument('--disable-logging')
        chrome_options.add_argument('--disable-extensions')
        chrome_options.add_argument('--disable-notifications')
        chrome_options.add_argument('--disable-default-apps')
        chrome_options.add_argument('--window-size=1920,1080')
        chrome_options.add_argument('--headless')
        chrome_options.add_argument('--no-sandbox')
        chrome_options.add_argument('--disable-dev-shm-usage')
        chrome_options.add_experimental_option('excludeSwitches', ['enable-automation'])
        chrome_options.add_experimental_option('useAutomationExtension', False)

        driver = webdriver.Chrome(executable_path=CHROME_DRIVER,
                                  desired_capabilities=dc,
                                  chrome_options=chrome_options)

        setattr(threadLocal, 'driver', driver)

    return driver

I read the logs using

    while True:
       logs = driver.get_log('browser')
       for log in logs:
          # do something

My initial guess was that the headless driver crashed when this occured. However, later in my script I have the following code which is satisfied (returned None):

if len(logs) == 0:
    try:
        if 'END' in driver.find_element(By.XPATH, f"(.//*[@class='sr-match-set-sports__status-str srm-is-uppercase']").text:
            return None

    except NoSuchElementException:
        continue

I assume that if the driver crashed this line should return a NoSuchElementException, so I can conclude that it did not crash?

I am also certain that additional logs should have been received by checking simultaneously the url in Google Chrome.

Any idea what's causing this behaviour?

CodePudding user response：

well I'm not sure but this might be because of traffic you are getting due to headless. try driver.implicitly_wait(5)