I want to write the prices of the products on this site with the code below with beautifulsoup, but when I write the code, the list returns empty.
import requests as req from bs4 import BeautifulSoup as bs
url = "https://www.migros.com.tr/temel-gida-c-2?sayfa=1"
headers = { 'User-Agent': ( 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) ' 'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36' ) }
urunler = []
rx = req.get(url, headers=headers) sympo = bs(rx.text, 'html.parser') soup = bs(rx.content, 'lxml') print(sympo.findAll('span', {'class': 'amount'}))
print(sympo.findAll('span', {'class': 'amount'}))
[]
CodePudding user response：
The page is generated by a JS call, so you're probably going to have to look into the selenium package. If you just check the output when you print soup, you'll only get the header content for the page:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<title>Migros Sanal Market: Online Market Alışverişi</title>
<base href="/"/>
<meta content="width=device-width, initial-scale=1, maximum-scale=5, user-scalable=0" name="viewport"/>
<link href="manifest.json" rel="manifest"/>
<link href="/assets/favicon.ico" rel="icon" type="image/x-icon"/>
<meta content="#ff9933" name="theme-color"/>
<meta content="Geniş ürün yelpazesi, en uygun fiyatlı ürünleri ve aynı gün teslimat fırsatıyla online market alışverişinizde Migros Sanal Market size iyi gelecek!" name="description" property="og:description"/>
<meta content="Migros Sanal Market: Online Market Alışverişi" property="og:site_name"/>
<meta content="https://migros-dali-storage-prod.global.ssl.fastly.net/sanalmarket/custom/sanalmarket-seo-34706362.png" property="og:image"/>
<link href="/assets/images/seo/apple-touch-icon-57x57.png" rel="apple-touch-icon" sizes="57x57"/>
<link href="/assets/images/seo/apple-touch-icon-114x114.png" rel="apple-touch-icon" sizes="114x114"/>
<link href="/assets/images/seo/apple-touch-icon-72x72.png" rel="apple-touch-icon" sizes="72x72"/>
<link href="/assets/images/seo/apple-touch-icon-144x144.png" rel="apple-touch-icon" sizes="144x144"/>
<link href="/assets/images/seo/apple-touch-icon-60x60.png" rel="apple-touch-icon" sizes="60x60"/>
<link href="/assets/images/seo/apple-touch-icon-120x120.png" rel="apple-touch-icon" sizes="120x120"/>
<link href="/assets/images/seo/apple-touch-icon-76x76.png" rel="apple-touch-icon" sizes="76x76"/>
<link href="/assets/images/seo/apple-touch-icon-152x152.png" rel="apple-touch-icon" sizes="152x152"/>
<link href="/assets/images/seo/apple-touch-icon.png" rel="apple-touch-icon"/>
<link href="/assets/images/seo/apple-touch-icon-57x57-precomposed.png" rel="apple-touch-icon-precomposed" sizes="57x57"/>
<link href="/assets/images/seo/apple-touch-icon-114x114-precomposed.png" rel="apple-touch-icon-precomposed" sizes="114x114"/>
<link href="/assets/images/seo/apple-touch-icon-72x72-precomposed.png" rel="apple-touch-icon-precomposed" sizes="72x72"/>
<link href="/assets/images/seo/apple-touch-icon-144x144-precomposed.png" rel="apple-touch-icon-precomposed" sizes="144x144"/>
<link href="/assets/images/seo/apple-touch-icon-60x60-precomposed.png" rel="apple-touch-icon-precomposed" sizes="60x60"/>
<link href="/assets/images/seo/apple-touch-icon-120x120-precomposed.png" rel="apple-touch-icon-precomposed" sizes="120x120"/>
<link href="/assets/images/seo/apple-touch-icon-76x76-precomposed.png" rel="apple-touch-icon-precomposed" sizes="76x76"/>
<link href="/assets/images/seo/apple-touch-icon-152x152-precomposed.png" rel="apple-touch-icon-precomposed" sizes="152x152"/>
<link href="/assets/images/seo/apple-touch-icon-precomposed.png" rel="apple-touch-icon-precomposed"/>
<script src="https://www.googleoptimize.com/optimize.js?id=OPT-K4CCSKT"></script>
<!-- To make possible for marketing partners to capture utm tags -->
<!-- For more info, check: https://developers.google.com/web/updates/2020/07/referrer-policy-new-chrome-default -->
<script charset="UTF-8" src="//cdn.segmentify.com/7a036bce-712d-4354-b5aa-2e74724eee62/segmentify.js"></script>
<meta content="no-referrer-when-downgrade" name="referrer"/>
<script src="env.js"></script>
<script async="" src="https://securepubads.g.doubleclick.net/tag/js/gpt.js"></script>
<link href="styles.0fb2916d96e6de1ada0e.css" rel="stylesheet"/></head>
<body >
<sm-root></sm-root>
<script defer="" src="runtime.8f34fe4ebfd1caac7fe2.js"></script><script defer="" src="polyfills.50ee7712f0e1700f8bcd.js"></script><script defer="" src="scripts.8129d19be5ffb616004f.js"></script><script defer="" src="main.3fec2e7e78a880c219a7.js"></script></body>
</html>
CodePudding user response：
The data you see is loaded via Javascript. You can use requests/json module to load it:
import json
import requests
url = "https://www.migros.com.tr/rest/search/screens/temel-gida-c-2?sayfa=1"
data = requests.get(url).json()
# uncomment this to print all data:
# print(json.dumps(data, indent=4))
for p in data["data"]["searchInfo"]["storeProductInfos"]:
print(
"{:<35} {:<10} {:<10}".format(
p["name"], p["regularPrice"], p["salePrice"]
)
)
Prints:
Maydanoz Adet 445 445
Soğan Kuru Dökme Kg 195 195
Migros Havuç Beypazarı Paket Kg 875 750
Domates Kg 1495 1495
Kabak Sakız Kg 1990 1990
Dereotu Adet 930 930
Roka Demet 603 603
Salata Kıvırcık Adet 1090 1090
Patlıcan Kemer Kg 1990 1990
Soğan Taze Demet 925 925
Hıyar Kg 1790 1790
Domates Salkım Kg 2290 2290
Biber Kırmızı Kg 2190 2190
Brokoli Kg 3450 3450
Atom Salata Adet 1206 1206
Kereviz Kg 875 875
Karnabahar Kg 1390 1390
Ispanak Kg 1450 1450
Patates Taze Kg 556 556
Biber Köy Usulü Kg 2990 2990
Nane Adet 631 631
Biber Sivri Kg 2690 2690
Pırasa Kg 930 930
Lahana Beyaz Kg 595 595
Biber Dolmalık Kg 2702 2702
Domates Şeker 250 G 837 837
Lahana Kırmızı Kg 1206 1206
Patates Ekonomik Boy File Kg 445 445
Pancar Kg 743 743
Domates Pembe Kg 1850 1850