When we use Selenium to test a page, such as using webdriver.Firefox() or webdriver.Chrome(), how does Selenium know when is the time that the page has finished rendering?

For example, if we do

browser = webdriver.Chrome()
browser.get(url)
result = browser.find_element_by_css_selector('#some-data')

I have noticed that if a page asynchronously fetches the data (using xhr) and displays it on the page, the last line in the code above was able to find the data.

However, what is the rule that browser.find_elements_by_css_selector() can proceed? It can be 1 second, 2 seconds, or the page can even have a setTimeout() after 2 seconds to fetch some data and fill in the page (say, if it is secondary important data). So what is the determining factor that Selenium knows: the page has finished rendering and we can find elements in it now?

CodePudding user response：

If you have requests asynchronously, selenium doesn't know when it's complete. But you can wait for an element and define how long you want to wait for it. Following the documentation https://selenium-python.readthedocs.io/waits.html

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))
    )
finally:
    driver.quit()

CodePudding user response：

Selenium knows: the page has finished rendering and we can find elements in it now?

Selenium WebDriver has inbuilt pageLoadStrategy. On launching a web page or navigating to a new page before it performs any actions it waits till document.readyState status changes to Complete

• If pageLoadStartegy is NORMAL. In this way, WebDriver waits till all dependent resources such as stylesheets and images loaded.

    setPageLoadStrategy(PageLoadStrategy.NORMAL);

• If pageLoadStrategy is EAGER. In this way, WebDriver waits till HTML content is loaded and parsed, without waiting for stylesheets, images, and subframes to finish loading.

    setPageLoadStrategy(PageLoadStrategy.EAGER);

Now there is another question for wait statements.

After page is loaded, we try to perform some action like click() get() etc. For an example, on clicking a button it displays a table of information in a web page within some time and we need access that information but to fetch that information it may take some time so here we can use wait statements.

Code#1: Using time.sleep() is not recommended.

• To fetch the information it may take around 6 seconds with this sleep statements driver waits till 10 seconds even data is available before the given time.

• Sometimes due to many reasons fetaching information takes more than 10 seconds in that case it fails.

    driver.find_element_by_css_selector("element").click()
    time.sleep(10)
    driver.find_element_by_xpath("element2").text

Code#2: Using Implicit wait.

• It is a global wait and applied to all elements on the webpage.
• Driver waits for 10 seconds before clicking element as well getting text from element2

    driver.implicitly_wait(10)
    driver.find_element_by_css_selector("element").click()
    driver.find_element_by_xpath("element2").text

Code#: Using Explict wait.

• It allows the code to halt program execution, or freeze the thread, until the condition you pass it resolves for given amount of time.
• If the condition is satisfied then the remaining wait time will be aborted.
• In below code, if the text is available at 6th second then rest of the wait time will be aborted.

    driver.find_element_by_css_selector("element").click()
    WebDriverWait(driver, 10).until(expected_conditions.visibility_of_element_located(
            (By.XPATH, "element2")))
    driver.find_element_by_xpath("element2").text

There are other wait functions like FluentWait and setScriptTimeOut also available.

References:

1. https://www.selenium.dev/documentation/webdriver/page_loading_strategy/
2. https://www.selenium.dev/documentation/webdriver/waits/