Why is BeautifulSoup not able to find some elements?-CodePudding

I am trying to read booking.com prices with python beautifulsoup, but some elements are missing. In this example I can read only the original price, but the discounted price can't be find with beautifulsoup.

Here HTML from Chrome:

and here what I get from bs4.

Any idea how I can get/reach the missing span element with the discounted price?

Here also the url from the webpage I wanted to scrap.

CodePudding user response：

try to search with particular class.. in this case soup.select(".d8etc"):

CodePudding user response：

The behaviour has nothing to do with beautifulsoup but depends on the response you receive to your request.

Such pages have a high dynamic for content and pricing depending on the current request (device, ip, browser, cookies, ...). It is therefore very likely that 10 different clients will receive 10 different results, and that the discount will possibly only be displayed in the context of a/b tests, ...

From my point of view, you should simply check whether the discount is available and, if in doubt, save it away.

...
data = []
for e in soup.select('[data-testid="property-card"]'):
    prices = e.select('[data-testid="price-and-discounted-price"] span')
    data.append({
        'title':e.h3.div.text,
        'price':prices[0].text,
        'price_discounted': prices[1].text if len(prices) > 1 else None
    })
data