I am new to web scraping and am trying to scrape data from this real estate website to get only the places that have recently been rented. To do this I need to click "Leased Listing" from this dropdown menu. Picture of what I need to click
The issue I am having is this is not a button class so using the selenium .click() function is giving me an error. There are also multiple objects with the same class name as the "Leased Listing" section.
Here is my code:
for page in range(0, total_pages 1):
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0 ; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36'}
url = 'https://www.zoocasa.com/toronto-on-real-estate-for-rent?page=' str(page)
driver.get(url)
elements = driver.find_elements(By.CLASS_NAME, "style_component__DR_Bs")
elements[6].click() #Leased listing is the 7th appearance of this class name
And here is the site's html (whatever is clicked has the "style_active__eGbvT"):
<div >
::before
Active Listing
::after
</div>
<div >
::before
Leased Listing
::after
</div>
<div >
::before
Expired Listing
::after
</div>
If anyone has any suggestions I would really appreciate it, Thanks.
CodePudding user response:
sorry cannot comment due to my reputation <50
I think selenium allows to click div
However, you need to identify the correct div
and id
before proceeding.
The easiest way, can you driver.find_elements_by_xpath
, and it will return a list then you need to check text
for each item
Code:
list_subcontent = driver.find_elements_by_xpath(".//div[@class='style_component__lT4sh style_theme-dark__rZF3s']//div[@class='style_component__DR_Bs ']")
for item in list_subcontent:
if item.text == 'Leased Listing':
item.click()
help to upvote if this help
CodePudding user response:
There are 2 problems here:
- You are trying to access the elements before the page is completely loaded. The best way to resolve this issue is to use expected conditions explicit waits to wait for some element visibility etc.
- The elements matching your locator are not clickable. You have to open the drop down menu to make that element clickable.
This should work better:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver.get(url)
wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.XPATH, "//div[contains(@class,'style_theme-dark')]//div[contains(text(),'Active Listings')]"))).click()
wait.until(EC.visibility_of_element_located((By.XPATH, "//div[contains(@class,'style_theme-dark')]//div[contains(text(),'Leased Listing')]"))).click()