I tried to scrape the price from the website below: https://www.emma-sleep.com.au/diamond-hybrid/ The price i wanna get should be 674.55.

Below is my code:

web_url = 'https://www.emma-sleep.com.au/diamond-hybrid/'
web_response = requests.get(web_url)
b_soup = BeautifulSoup(web_response.text, 'html.parser')
price=b_soup.find_all('span', {'class': 'installment__full-price'})
price

it will return me of a price of 1000 somehow, which is wrong [ $1,000.00 ]

could someone please tell me where i get it wrong? seems like the price is in the 'script' tage when i checked the html code.

Thanks :)

CodePudding user response：

The desired minimal working solution where html dom is <span data-price="674.55">$674.55</span> and the desired output is price $674.55

Code

from bs4 import BeautifulSoup
import time
from selenium import webdriver

driver = webdriver.Chrome('chromedriver.exe')
driver.maximize_window()
time.sleep(8)

url = 'https://www.emma-sleep.com.au/diamond-hybrid/'
driver.get(url)
time.sleep(5)

soup = BeautifulSoup(driver.page_source, 'lxml')
price=soup.select_one('div.cell.small-6 span:nth-child(2)').text
print(price)

Output

$674.55

CodePudding user response：

The price is calculated in javascript. For that you need to use an API like Selenium that first renders the page in the browser then allows you to access the HTML.

Target html tags:

<span class="regular-price text-orange" id="product-price-3853">
   <span class="price" data-price="674.55">$674.55</span>
</span>

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

web_url = 'https://www.emma-sleep.com.au/diamond-hybrid/'

s = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=s)

# web driver goes to page
driver.get(web_url)

# to give time for the page to load
driver.implicitly_wait(3)

price = driver.find_element(By.XPATH, "//span[@class='regular-price text-orange']/span[@class='price']").text
print(price)

Output:

$674.55