How to get prices from html by python?-CodePudding

Im trying to web scrape The North Face, I managed to get the title, description, color, ecc. But seems to hard for me to get the prices. How can i get the price of a product?

I tried something like selenium but didnt work, I got an empty string. Someone know how can I do this stuff?

import requests
from bs4 import BeautifulSoup as bs



r = requests.get("https://www.thenorthface.it/shop/it/tnf-it/berretto-jim-a5wh?variationId=KS7")
contenuto = bs(r.text, "html.parser")



title = contenuto.find(class_='product-content-info-name product-info-js').text.strip()
print(title)

descrizione = contenuto.find(class_='desc-container pdp-details-desc-container').text.strip()
print(descrizione)

caratteristiche = contenuto.find(class_='inner-content product-details-section-inner-content pdp-features-inner-content').text.strip()
print(caratteristiche)

colore = contenuto.find(class_='product-content-form-attr-selected attr-selected attribute-label-value attribute-label-value-js').text.strip()
print(colore)

prezzo= contenuto.find(class_='product-price price')
print(prezzo)

CodePudding user response：

I was always doing it with "for" loop. So it looks like this:

for i in contenuto.find_all("span", class_="product-content-info-name product-info-js"):
     title=i.get_text()

And it is working. Try it and give me feedback.

CodePudding user response：

Main issue here is that the content is provided dynamically by JavaScript as one of the class names hints.

So there are various ways to get a price, depending on what is needed exactly:

Check the static content of meta:

contenuto.select_one('[property="og:price:amount"]').get('content')

Check the static price section :

contenuto.select_one('[itemprop="price"]').get('content')

Check meta again and request the actually price for specific store via ajax price url:
```
contenuto.select_one('[property="og:price:url"]').get('content')
```

Check the JavaScript variables and extract the price from there:

var itemPrices = {"1267185":{"pricing":{"productId":"1267185","default":{"lowPrice":"€ 112,00","lowPriceNumeric":112,"highPrice":"€ 160,00","highPriceNumeric":160,"lowListPrice":"€ 160,00","lowListPriceNumeric":160,"highListPrice":"€ 160,00","highListPriceNumeric":160},"7741124012283612344":{"8W8":{"highPrice":"€ 160,00","highPriceNumeric":160,"lowPrice":"€ 160,00","lowPriceNumeric":160,"highListPrice":"€ 160,00","highListPriceNumeric":160,"lowListPrice":"€ 160,00","lowListPriceNumeric":160,"sku":[1295427,1295426,1295430,1295429,1295428]},"43M":{"highPrice":"€ 160,00","highPriceNumeric":160,"lowPrice":"€ 160,00","lowPriceNumeric":160,"highListPrice":"€ 160,00","highListPriceNumeric":160,"lowListPrice":"€ 160,00","lowListPriceNumeric":160,"sku":[1267195,1267199,1267198,1267197,1267196]},"5R7":{"highPrice":"€ 112,00","highPriceNumeric":112,"lowPrice":"€ 112,00","lowPriceNumeric":112,"highListPrice":"€ 160,00","highListPriceNumeric":160,"lowListPrice":"€ 160,00","lowListPriceNumeric":160,"sku":[1267203,1267202,1267201,1267200,1267204]}},"7741124012283612349":{"XL":{"highPrice":"€ 160,00","highPriceNumeric":160,"lowPrice":"€ 112,00","lowPriceNumeric":112,"highListPrice":"€ 160,00","highListPriceNumeric":160,"lowListPrice":"€ 160,00","lowListPriceNumeric":160,"sku":[1267203,1295429,1267198]},"L":{"highPrice":"€ 160,00","highPriceNumeric":160,"lowPrice":"€ 112,00","lowPriceNumeric":112,"highListPrice":"€ 160,00","highListPriceNumeric":160,"lowListPrice":"€ 160,00","lowListPriceNumeric":160,"sku":[1295426,1267200,1267195]},"XS":{"highPrice":"€ 160,00","highPriceNumeric":160,"lowPrice":"€ 112,00","lowPriceNumeric":112,"highListPrice":"€ 160,00","highListPriceNumeric":160,"lowListPrice":"€ 160,00","lowListPriceNumeric":160,"sku":[1295430,1267204,1267199]},"S":{"highPrice":"€ 160,00","highPriceNumeric":160,"lowPrice":"€ 112,00","lowPriceNumeric":112,"highListPrice":"€ 160,00","highListPriceNumeric":160,"lowListPrice":"€ 160,00","lowListPriceNumeric":160,"sku":[1267202,1295428,1267197]},"M":{"highPrice":"€ 160,00","highPriceNumeric":160,"lowPrice":"€ 112,00","lowPriceNumeric":112,"highListPrice":"€ 160,00","highListPriceNumeric":160,"lowListPrice":"€ 160,00","lowListPriceNumeric":160,"sku":[1295427,1267201,1267196]}},"sku":{"1295428":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 160,00","priceNumeric":160},"1295427":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 160,00","priceNumeric":160},"1295426":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 160,00","priceNumeric":160},"1267204":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 112,00","priceNumeric":112},"1267203":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 112,00","priceNumeric":112},"1267202":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 112,00","priceNumeric":112},"1267201":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 112,00","priceNumeric":112},"1267200":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 112,00","priceNumeric":112},"1267199":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 160,00","priceNumeric":160},"1267198":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 160,00","priceNumeric":160},"1267197":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 160,00","priceNumeric":160},"1267196":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 160,00","priceNumeric":160},"1267195":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 160,00","priceNumeric":160},"1295430":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 160,00","priceNumeric":160},"1295429":{"listPrice":"€ 160,00","listPriceNumeric":160,"price":"€ 160,00","priceNumeric":160}}}}};