BeautifulSoup not working on this website-CodePudding

from bs4 import BeautifulSoup
import requests

# HTML From Website
url = "https://myaeon2go.com/product/11073/local-tomato"

#convert soup to readable html
result = requests.get(url)
doc = BeautifulSoup(result.text, "html.parser")

prices = doc.find_all("span", {"class": "oVW2jqeJPDfka7D8UVAd"})
rm = doc.find_all(text="RM")

print(prices)

Ive been trying to wrap my head around this code, but even if i tried looking for other elements the classes are all randomly generated. The code just outputs [] . Not sure why that is?

CodePudding user response：

Problem is modal window (location overlay) with input of postal code which pop-up when script request the page. You can pass cookies which suppress window when requesting like this:

...
cookies = {'hideLocationOverlay': 'true', 'selectedShippingState': 'Pulau Pinang', 'selectedPostalCode': '14000'}
    
#convert soup to readable html
result = requests.get(url, cookies=cookies)
doc = BeautifulSoup(result.text, "html.parser")

...

To find the price I would recommend this approach:

price = doc.find("div", {"data-bx": "pdp-details-price"}).find("span").find("span").text

print(price)