find_elements not finding all elements-CodePudding

I'm trying to find all the elements on a page with the specific selector but as it seems, not all the elements are found.

Code I'm using:

from selenium import webdriver
from selenium.webdriver.common.by import By

PATH = "C:\Program Files (x86)\chromedriver.exe"#path of chrome driver
driver = webdriver.Chrome(PATH)#accesses the chrome driver

web = driver.get("https://www.eduqas.co.uk/qualifications/computer-science-as-a-level/#tab_pastpapers")#website
driver.maximize_window()

driver.implicitly_wait(3)
driver.execute_script("window.scrollTo(0, 540)")
driver.implicitly_wait(3)
elements = driver.find_elements(By.CSS_SELECTOR, ".css-13punl2")
driver.implicitly_wait(3)

for x in elements:
    x.click()


print(len(elements))

When I print the length of the array "elements" it returns 1, when there are multiple elements on the web page with the selector ".css-13punl2". As seen here image of web page code

link to the website: https://www.eduqas.co.uk/qualifications/computer-science-as-a-level/#tab_pastpapers

For some reason, when I inspect the web page there will sometimes be 6 elements with selector ".css-13punl2" and sometimes there will be 7, but I'm not too sure.

CodePudding user response：

is the selector stable? im not much familiar with selenium in python, but from what i know, in runtime there are some element attributes that change...

my advice: put a sleep for 30 seconds, open console (F12) in the opened driver, and write the following command:

$$(".css-13punl2")

if it gives you only 1 element, than you found the problem or that even it gave you 6 elements, but most of them are invisible.

could you also provide a screenshot of the web itself? or even the link to it

EDITED ANSWER:

try this selector:

#pastpapers_content button

CodePudding user response：

Don't use implicitly_wait() more than once, try the below code, it will click on each year:

driver.get("https://www.eduqas.co.uk/qualifications/computer-science-as-a-level/#tab_pastpapers")
wait = WebDriverWait(driver, 10)
time.sleep(1)
# to click on Accept Cookies button
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#accept-cookies"))).click()

# waiting for list of years to appear and scrolling to it
content = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "#pastpapers_content")))
driver.execute_script("arguments[0].scrollIntoView(true)", content)

# get all the elements
elements = wait.until(EC.presence_of_all_elements_located((By.XPATH, ".//*[@id='pastpapers_content']//button[@type='button']")))
print("Total elements: ", len(elements))

for i in range(len(elements)):
    ele = driver.find_element(By.XPATH, "(.//*[@id='pastpapers_content']//button[@type='button'])["   str(i   1)   "]")
    time.sleep(1)
    ele.location_once_scrolled_into_view
    time.sleep(1)
    ele.click()

CodePudding user response：

You are trying to wait for the element with driver.implicitly_wait(3). This method is created for activating implicit wait. It can be turned On once and then you don't need to turn it again. When it's turned On Selenium will try to find the element you want for 3 (in this case) seconds instead of one attempt immediately after page loading. In your case it finds one element quickly and doesn't wait for all the elements to appear on the page.
You need to give the page some time to load all the stuff. For that you can use a sleep method for example. It pauses the script execution for the amount of seconds you've set.
Also, the elements on this page disappear from the view so you need to scroll each time you click an element. Additionally you need to close this 'Cookie' prompt as it intercepts the clicks.
And I guess you need the elements with Year, so it's better to skip clicking the 'GCSE' which is also found by this selector.
So, the code will look like that:

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep

PATH = "C:\Program Files (x86)\chromedriver.exe"  # path of chrome driver
driver = webdriver.Chrome(PATH)  # accesses the chrome driver

driver.get("https://www.eduqas.co.uk/qualifications/computer-science-as-a-level/#tab_pastpapers")  # website
driver.maximize_window()

driver.implicitly_wait(3)
driver.execute_script("window.scrollTo(0, 540)")
sleep(3)  # Giving time to fully load the content
elements = driver.find_elements(By.CSS_SELECTOR, ".css-13punl2")
driver.find_element(By.ID, 'accept-cookies').click()  # Closes the cookies prompt


for x in elements:
    if x.text == 'GCSE':
        continue
    x.click()
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")  # this scrolls the page to the bottom
    sleep(1)  # This sleep is necessary to give time to finish scrolling


print(len(elements))