In python I wrote:

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)
driver.get(url)
try:
    WebDriverWait(driver, 10).until(
        lambda driver: driver.execute_script('return document.readyState') == 'complete')
except se.TimeoutException:
    return False
# Start Parsing

Even though I have waited for readyState for some websites when I parse it I see that there is no checkbox. But, If I add time.sleep(5) Before parsing for the same website I get that there is a checkbox.

My question is, how can I have a general solution that works with the majority of websites? I can't just write time.sleep(5) as some websites might need much more and some might finish within 0.001 seconds (which will have bad impact on performance...)

I just want to stimulate a real browser and not to handle anything before the refresh button appears again (which means everything was loaded).

CodePudding user response：

Ideally web applications when accessed through get(), returns the control to the WebDriver only when document.readyState equals to complete. So unless the AUT(Application under Test) behaves otherwise, the following line of code is typically an overhead:

WebDriverWait(driver, 10).until(lambda driver: driver.execute_script('return document.readyState') == 'complete')

However, as per your test requirements you can configure the pageLoadStrategy either as:

• none
• eager
• normal

You can find a detailed discussion in What is the correct syntax checking the .readyState of a website in Selenium Python

At this point, it is to be noted that using time.sleep(secs) without any specific condition to achieve defeats the purpose of Automation and should be avoided at any cost.

Solution

The generic approach that would work with all the websites is to induce WebDriverWait as per the prevailing test scenario. As an example:

• To wait for the presence of an element you need to invoke the expected_conditions of presence_of_element_located()
• To wait for the visibility of an element you need to invoke the expected_conditions of visibility_of_element_located()
• To wait for the element to be visible, enabled and interactable such that you can click it you need to invoke the expected_conditions of element_to_be_clickable()