I am new to web scraping and am trying to scrape data from this real estate website to get only the places that have recently been rented. To do this I need to click "Leased Listing" from this dropdown menu. Picture of what I need to click

The issue I am having is this is not a button class so using the selenium .click() function is giving me an error. There are also multiple objects with the same class name as the "Leased Listing" section.

Here is my code:

 for page in range(0, total_pages   1):
  chrome_options = webdriver.ChromeOptions()
  chrome_options.add_argument('--headless')
  chrome_options.add_argument('--no-sandbox')
  chrome_options.add_argument('--disable-dev-shm-usage')
  driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
  headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0 ; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36'}
  url = 'https://www.zoocasa.com/toronto-on-real-estate-for-rent?page='   str(page)
  driver.get(url)
  elements = driver.find_elements(By.CLASS_NAME, "style_component__DR_Bs")
  elements[6].click() #Leased listing is the 7th appearance of this class name

And here is the site's html (whatever is clicked has the "style_active__eGbvT"):

<div >
    ::before
    Active Listing
    ::after
</div>
<div >
    ::before
    Leased Listing
    ::after
</div>
<div >
    ::before
    Expired Listing
    ::after
</div>

If anyone has any suggestions I would really appreciate it, Thanks.

CodePudding user response：

sorry cannot comment due to my reputation <50

I think selenium allows to click div

However, you need to identify the correct div and id before proceeding.

The easiest way, can you driver.find_elements_by_xpath, and it will return a list then you need to check text for each item

Code:

list_subcontent = driver.find_elements_by_xpath(".//div[@class='style_component__lT4sh style_theme-dark__rZF3s']//div[@class='style_component__DR_Bs ']")

for item in list_subcontent:
    if item.text == 'Leased Listing':
        item.click()

help to upvote if this help

CodePudding user response：

There are 2 problems here:

1. You are trying to access the elements before the page is completely loaded. The best way to resolve this issue is to use expected conditions explicit waits to wait for some element visibility etc.
2. The elements matching your locator are not clickable. You have to open the drop down menu to make that element clickable.
   This should work better:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver.get(url)
wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.XPATH, "//div[contains(@class,'style_theme-dark')]//div[contains(text(),'Active Listings')]"))).click()
wait.until(EC.visibility_of_element_located((By.XPATH, "//div[contains(@class,'style_theme-dark')]//div[contains(text(),'Leased Listing')]"))).click()