Selenium find_elements only works if time passes since get()-CodePudding

Running these 2 lines together in Colab returns []:

wd.get("https://wetransfer.com/")
wd.find_elements(By.CSS_SELECTOR, 'input[type=file]')

However, running one, followed by the other returns the expected result:

[<selenium.webdriver.remote.webelement.WebElement (session="3cdfb3afbb591862e909cd406b6ac523", element="19fd31e8-710a-4b6e-8284-9a7409f12718")>,
<selenium.webdriver.remote.webelement.WebElement (session="3cdfb3afbb591862e909cd406b6ac523", element="837097d1-5735-4b24-9cb2-9d3ded3a0311")>]

Get is supposed to be blocking so not sure what is going on here.

CodePudding user response：

This is how basically Selenium works.
It can access web elements only after they are completely loaded.
This is why we use implicitly and explicitly waits here.
The explicitly waits are much more recommended.
So instead of

wd.get("https://wetransfer.com/")
wd.find_elements(By.CSS_SELECTOR, 'input[type=file]')

You should use something like this:

wd.get("https://wetransfer.com/")
wait = WebDriverWait(wd, 20)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "input[type=file]")))
wd.find_elements(By.CSS_SELECTOR, 'input[type=file]')

To use it you will have to import these imports:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

CodePudding user response：

You can use the following code as an example, to wait until your the site is loaded completly and ready for manipulations.

myElem = WebDriverWait(driver, delay).until(EC.element_to_be_clickable((By.CLASS_NAME , 'bs_btn_buchen')))

CodePudding user response：

On invoking get() as soon as the browser sends 'document.readyState' == "complete" to the driver, Colab executes the next line of code which doesn't find any match as the DOM Tree have completely not loaded. Hence you see Colab returning []

To locate the visible elements you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "input[type=file]")))

Using XPATH:

elements = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//input[@type=file]")))

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC