Choosing from a Dropdown in Selenium--Element is not visible (Python)-CodePudding

I'm trying to scrape data about soccer players on this website: https://understat.com/team/Aston_Villa/2021

This specific page is the page of Aston Villa. The thing is that I need to scrape data from only the 'Last 5 Games'. This is a dropdown about half-way down the webpage of the link I attached. In order to change this from 'All' to '5 games', I need to click on the dropdown, select '5 games', and then click on the filter button.

I've tried selecting the option '5 games', but the element is not visible without clicking on it first, and I wouldn't know how to click on the Filter button afterwards anyway.

I've tried select = Select(driver.find_element(By.NAME, 'team-players-n-last-matches')) select.select_by_value('5')

and I've tried

driver.find_element_by_xpath("//select[@name='team-players-n-last-matches']/option[text()='5 games']").click()

But neither one of these works because A) the element is still invisible, and B) these options don't click on the filter button afterwards.

In summary, I need to:

A) click on the 'All' dropdown halfway down the page

B) choose the '5 games' option

C) click on the filter button to apply the filter

Thank you to anyone who can help in advance!!

Adam

CodePudding user response：

It's tricky since the dropdown has custom styles and bindings.

Here's a way you can get what you want

driver.execute_script('document.evaluate("//div[text()=\'All games\']", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue?.click()')

driver.execute_script('document.querySelector(\'[rel="5"]\').click()')

driver.execute_script('document.querySelector(".fa.fa-filter").click()')

I used the word 'All games' to identify the clickable div for the dropdown you target by using xpath, then used rel attribute to select the list-item that has 5 games, then used .fa.fa-filter classes to click on the filter button

CodePudding user response：

To bypass the element visibility requirement of driver.find_element, you can use driver.execute_script and run custom Javascript instead:

from selenium import webdriver
d = webdriver.Chrome('/path/to/chromedriver')
d.get('https://understat.com/team/Aston_Villa/2021')
d.execute_script("""
   document.querySelector('div.block:nth-of-type(4) div.filter:nth-of-type(2) .custom-select-styled').click()
   document.querySelector('div.block:nth-of-type(4) div.filter:nth-of-type(2) li[rel="5"]').click()
   document.querySelector('button#team-players-filter').click()
""")