HTML from site not the same as the returned beautiful soup object-CodePudding

I am trying to scrape ubereats for prices with Beasutiful soup. However, the HTML from the beautiful soup object is not the same with the ubereats HTML when i check the source code from the developer tools in windows. When i try to get the product name, i don't receive any results. I am reading now that Beautiful soup can only pull HTML, and maybe the site is behind JS. Is there any way to work this around but still use beautiful soup? Thank you all

import requests
import bs4
from bs4 import BeautifulSoup

ubereats_url = ubereats_url = 'https://www.ubereats.com/gb/store/londis-cleadon/nvDm7CwVTBeNK2MzAK6sOQ?diningMode=DELIVERY&pl=JTdCJTIyYWRkcmVzcyUyMiUzQSUyMk5FMzQlMjA4QVElMjIlMkMlMjJyZWZlcmVuY2UlMjIlM0ElMjJDaElKOFY2cDRrOXZma2dSZGNTVmlGTDdhZm8lMjIlMkMlMjJyZWZlcmVuY2VUeXBlJTIyJTNBJTIyZ29vZ2xlX3BsYWNlcyUyMiUyQyUyMmxhdGl0dWRlJTIyJTNBNTQuOTYzNDA4MSUyQyUyMmxvbmdpdHVkZSUyMiUzQS0xLjQxMzMyMDElN0Q='

plain_html_text = requests.get(ubereats_url)
soup = BeautifulSoup(plain_html_text.text, "lxml")
for product in soup.find_all('div', class_ = 'cg d1 ci d2 ax'):
    print(product.text)

CodePudding user response：

Try:

import requests
from bs4 import BeautifulSoup

ubereats_url = "https://www.ubereats.com/gb/store/londis-cleadon/nvDm7CwVTBeNK2MzAK6sOQ?diningMode=DELIVERY&pl=JTdCJTIyYWRkcmVzcyUyMiUzQSUyMk5FMzQlMjA4QVElMjIlMkMlMjJyZWZlcmVuY2UlMjIlM0ElMjJDaElKOFY2cDRrOXZma2dSZGNTVmlGTDdhZm8lMjIlMkMlMjJyZWZlcmVuY2VUeXBlJTIyJTNBJTIyZ29vZ2xlX3BsYWNlcyUyMiUyQyUyMmxhdGl0dWRlJTIyJTNBNTQuOTYzNDA4MSUyQyUyMmxvbmdpdHVkZSUyMiUzQS0xLjQxMzMyMDElN0Q="

plain_html_text = requests.get(ubereats_url)
soup = BeautifulSoup(plain_html_text.text, "lxml")

for t in soup.select("li:has(.lazyload-wrapper):not(:has(li))"):
    data = [s.text for s in t.select("span")]
    if len(data) != 2:
        continue
    print("{:<10} {}".format(data[1], data[0]))

Prints:


...

£2.49      Pot Noodle Beef &Tomato 90g
£2.49      Pot Noodle Original Curry 90g
£1.99      Tunnocks Chocolate Caramel Wafer 4x30g
£3.59      Cadbury Fingers 114g
£2.19      Oreo Cookies 154g
£4.29      McVities Blissfuls Chocolate & Caramel 228g
£2.69      McVities Jaffa Cakes 80% Extra Free 18 pack
£1.59      Maryland Choc Chip Cookie 200g PM
£2.49      McVities Milk Chocolate Hobnobs 262g PM
£2.49      McVities Milk Chocolate Digestives  266g PM
£4.69      Bonne Maman Madeleine 175g
£3.69      Cadburys Mini Roll Chocolate 5pk

...