I am not able to get to the div within so many nested divs. . . .

<div class="list-card-price">$2,600/mo</div>

. . .

I am using this method to retrive it:

from bs4 import BeautifulSoup
import requests
import lxml

response = requests.get(url="https://www.zillow.com/homes/for_rent/1-_beds/?.....")
data = response.text

soup = BeautifulSoup(data,"lxml")

price_list = []

price_tag = soup.select_one(name='div',class_="list-card-price")
print(price_tag)

CodePudding user response：

What happens?

1. Always take a deeper look into your soup - There is the truth! It comes up with a captcha warning.

2. Your selection will throw an error cause you mix syntax of find() and select_one()

How to fix?

1. Take a closer look at this answer options to deal with captcha

2. Change your selection to soup.select_one('div.list-card-price') to avoid the error TypeError: select_one() missing 1 required positional argument: 'selector'

CodePudding user response：

You can check the HTML that's being sent in the soup with "soup.text". When I ran the code, the website presented a captcha page to the requests module, not the actual webpage.