Home > Mobile >  Web scrapping: Why my code does not print the desired output?
Web scrapping: Why my code does not print the desired output?

Time：07-19

I'm working on my web scraping project using python language and BeautifulSoup. I tried the sample code below and the execute the desired output.

from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as uReq

URL = "https://www.newegg.com/Monitors/Category/ID-19?Tpk=monitor"
uClient = uReq(URL)
page_html = uClient.read()

page_soup = soup(page_html, "html.parser")
uClient.close()

containers = page_soup.findAll("div", {"class": "item-container"})
print(containers)

However, when I adjusted the code (link and html tag parts), the output is []. Is it because of the link or because of the html tag?

from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as uReq

URL = "https://www.lotuss.com.my/en/category/grocery"
uClient = uReq(URL)
page_html = uClient.read()

page_soup = soup(page_html, "html.parser")
uClient.close()

containers = page_soup.findAll("div",{"class":"product-grid-item"})
print(containers)

CodePudding user response：

The issue seems to be that the product-grid-item is not returned in the request you are making. It looks like they are using react so it's most likely being generated client side and all your request is getting is the index.html file that is used as the root for react.

Your best bet is to use something like Selenium: https://theleakycauldronblog.com/blog/scraping-dynamic-single-page-websites-with-scrapy-and-selenium

CodePudding user response：

The item is received dynamically and you can't get it with bs4. You must find a query that will return the items you need. After you study the request structure, you will have another question, how to get an access token. Simple:

def get_token():
    url = "https://identity-service-online-ecommerce-my-prod.auth.ap-southeast-1.amazoncognito.com/oauth2/token"
    payload = 'grant_type=client_credentials&scope=service-api/general'
    headers = {
      'authority': 'identity-service-online-ecommerce-my-prod.auth.ap-southeast-1.amazoncognito.com',
      'accept': 'application/json, text/plain, */*',
      'authorization': 'Basic M200YjE4bDhvZ3NhM2s0dXV0cDFxZ3VuZTg6MW4xaXJkMGJoZ2dzbGZlMTFiM3NsYWl0YjVyN3BjZG4xcDhqYWJsNHZiaGJtZTRibHVmYQ==',
      'content-type': 'application/x-www-form-urlencoded',
      'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
    }

    response = requests.request("POST", url, headers=headers, data=payload)
    return response.json()['access_token']

Now we can put the necessary parameters into the function, the first thing that comes to mind is offset, token and categories where we are looking for products. Offset is required for paging products. Since I looked at the url that you attached, we are interested in Grocery, which has the ID 2730. For example, I will display 90 products, specifying sku, name, price and url in each of them. Full code:

import requests
import json


def get_token():
    url = "https://identity-service-online-ecommerce-my-prod.auth.ap-southeast-1.amazoncognito.com/oauth2/token"
    payload = 'grant_type=client_credentials&scope=service-api/general'
    headers = {
      'authority': 'identity-service-online-ecommerce-my-prod.auth.ap-southeast-1.amazoncognito.com',
      'accept': 'application/json, text/plain, */*',
      'authorization': 'Basic M200YjE4bDhvZ3NhM2s0dXV0cDFxZ3VuZTg6MW4xaXJkMGJoZ2dzbGZlMTFiM3NsYWl0YjVyN3BjZG4xcDhqYWJsNHZiaGJtZTRibHVmYQ==',
      'content-type': 'application/x-www-form-urlencoded',
      'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
    }

    response = requests.request("POST", url, headers=headers, data=payload)
    return response.json()['access_token']


def show_items(token, offset, category_id):
    url = "https://shoponline-bffapi.lotuss.com.my/bff/product/v1/products?websiteCode=malaysia_hy"

    payload = json.dumps({
        "offset": offset,
        "limit": 30,
        "filter": {
            "categoryId": category_id
        }
    })
    headers = {
        'accept': 'application/json, text/plain, */*',
        'accept-language': 'en',
        'authorization': token,
        'content-type': 'application/json',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
    }
    response = requests.request("POST", url, headers=headers, data=payload)
    for product in response.json()['data']['products']:
        price = str(product['priceRange']['minimumPrice']['finalPrice']['value'])   product['priceRange']['minimumPrice']['finalPrice']['currency']
        url = 'https://www.lotuss.com.my/en/product/'   product['urlKey']
        print(product['sku'], product['name'], price, url)


access_token = get_token()
for page_offset in range(30, 120, 30):
    show_items(access_token, page_offset, ["2730"])

OUTPUT:

74697196 LOTUSS GREEN PEA SNACK 60G 1.09MYR https://www.lotuss.com.my/en/product/lotuss-green-pea-snack-60g-74697196
79170 SAN REMO NO.5 SPAGHETTI 500G 4.95MYR https://www.lotuss.com.my/en/product/san-remo-no-5-spaghetti-500g-79170
510637 CHACHEER SUNFLOWER SEEDS 220G 5.99MYR https://www.lotuss.com.my/en/product/chacheer-sunflower-seeds-220g-510637
2729881 TAO KAE NOI CRISPY SEAWEED ORIG 32.5G 6.59MYR https://www.lotuss.com.my/en/product/tao-kae-noi-crispy-seaweed-orig-32-5g-2729881
71138730 POCKY CHOCOLATE FLAVOUR 40G 2.95MYR https://www.lotuss.com.my/en/product/pocky-chocolate-flavour-40g-71138730
72527811 JACOBS CREAM CRACKER 700G 13.5MYR https://www.lotuss.com.my/en/product/jacobs-cream-cracker-700g-72527811
72338458 CED NATURAL HIMALAYAN PINK ROCK SALT 2.29MYR https://www.lotuss.com.my/en/product/ced-natural-himalayan-pink-rock-salt-72338458
74698990 Lotus's Ridge Cut Original Potato Chips 70g 2.19MYR https://www.lotuss.com.my/en/product/lotuss-potato-chips-r-cut-70g-original-74698990
243108 COCON PUDDING LYCHEE   FRIENDS 12X80G 11.99MYR https://www.lotuss.com.my/en/product/cocon-pudding-lychee- -friends-12x80g-243108
1363964 CED ORGANIC CANE SUGAR 500G 5.59MYR https://www.lotuss.com.my/en/product/ced-organic-cane-sugar-500g-1363964
3788679 E.S BRAND PICKLED LETTUCE 182G 2.99MYR https://www.lotuss.com.my/en/product/e-s-brand-pickled-lettuce-182g-3788679
2220679 SUPER RING CHEESE FAMILY PACK 8X14G 2.79MYR https://www.lotuss.com.my/en/product/super-ring-cheese-family-pack-8x14g-2220679
3341844 MUNCHYS OATKRUNCH DARK CHOC 390G 7.99MYR https://www.lotuss.com.my/en/product/munchys-oatkrunch-dark-choc-390g-3341844
3267024 FRASER HILLS HONEY 350G 6.71MYR https://www.lotuss.com.my/en/product/fraser-hills-honey-350g-3267024
3263746 BERASLI INSTANT PORRIDGE CHICKEN 40G 1.95MYR https://www.lotuss.com.my/en/product/berasli-instant-porridge-chicken-flavoured-40g-3263746
3242846 SNEKKU FAMILY PACK-TAM TAM 8X22G 2.99MYR https://www.lotuss.com.my/en/product/snekku-family-pack-tam-tam-8x25g-3242846
74138936 PRAI COARSE GRAIN SUGAR 1KG 2.85MYR https://www.lotuss.com.my/en/product/prai-coarse-grain-sugar-1kg-74138936
74690248 LOTUSS BUNGA CENGKIH 50G 4.69MYR https://www.lotuss.com.my/en/product/lotuss-bunga-cengkih-50g-74690248
1364901 A1 SOUP SPICES 35G 8.75MYR https://www.lotuss.com.my/en/product/a1-bak-kut-teh-soup-spices-35g-1364901
557285 KARA COCONUT CREAM 200ML 3.19MYR https://www.lotuss.com.my/en/product/kara-coconut-cream-200ml-557285
125733 NESTLE KOKO KRUNCH ECONOPACK 500G 14.99MYR https://www.lotuss.com.my/en/product/nestle-koko-krunch-econopack-500g-125733
72338474 CED ORG REGULAR ROLLED OATS 500G 7.99MYR https://www.lotuss.com.my/en/product/ced-org-regular-rolled-oats-500g-72338474
70655871 ADELA TUB  MARGARINE 1 KG 11.09MYR https://www.lotuss.com.my/en/product/adela-multi-purpose-margarine-1kg-70655871
4638255 JULIES OAT 25 TEN GRAINS 300G 8.05MYR https://www.lotuss.com.my/en/product/julies-oat-25-ten-grains-300g-4638255
4411757 MAGGI 2MIN CHICKEN FLAVOR 5X77G 3.99MYR https://www.lotuss.com.my/en/product/maggi-2min-chicken-flavor-5x77g-4411757
4350197 DAISY PEANUT BUTTER (CRUNCHY) 500G 11.79MYR https://www.lotuss.com.my/en/product/daisy-peanut-butter-crunchy-500g-4350197
2959216 SMUCKERS SUGAR FREE STRAWBERRY 361G 11.59MYR https://www.lotuss.com.my/en/product/smuckers-sugar-free-strawberry-361g-2959216
2252651 MENTOS MINT POUCH 36S (97.2G) 2.79MYR https://www.lotuss.com.my/en/product/mentos-mint-pouch-36s-97-2g-2252651
2605791 TUMIX CHICKEN FLAVOR 1KG 17.99MYR https://www.lotuss.com.my/en/product/tumix-chicken-flavor-1kg-2605791
897892 DOUBLE SWALLOW BARLI 100G 0.79MYR https://www.lotuss.com.my/en/product/double-swallow-barli-100g-897892
116122 ADABI SERBUK BIHUN SUP 20G 1.49MYR https://www.lotuss.com.my/en/product/adabi-serbuk-bihun-sup-20g-116122
553700 TAMIN TAUCU 450G 3.64MYR https://www.lotuss.com.my/en/product/tamin-taucu-450g-553700
512176 VOCHELLE DOYPK ASSORTMENT 100G 6.39MYR https://www.lotuss.com.my/en/product/vochelle-doypk-assortment-100g-512176
412260 YEOS BAKED BEAN IN TMT SOS 425G 4.45MYR https://www.lotuss.com.my/en/product/yeos-baked-bean-in-tmt-sos-425g-412260
73925446 LE DRIED MULBERRY 140G 13.99MYR https://www.lotuss.com.my/en/product/le-dried-mulberry-140g-73925446
74690582 LOTUSS BLACK RAISIN 120G 3.99MYR https://www.lotuss.com.my/en/product/lotuss-black-raisin-120g-74690582
414700 ROTA SUPER RING 60G 1.99MYR https://www.lotuss.com.my/en/product/rota-super-ring-60g-414700
1446169 BOP PHKARKHNEI NATURAL WHITE RICE 5KG 38.79MYR https://www.lotuss.com.my/en/product/bop-phkarkhnei-natural-white-rice-5kg-1446169
4358996 KNORR CUP PORRIDGE CHIC MUSH 35G 2.9MYR https://www.lotuss.com.my/en/product/knorr-cup-porridge-chic mush-35g-4358996
72830735 CAP LILI FORTIFIED WHEAT FLOUR 850G 2.35MYR https://www.lotuss.com.my/en/product/cap-lili-fortified-wheat-flour-850g-72830735
74700308 LOTUSS CHIPS 110G SOUR CREAM ONION 2.49MYR https://www.lotuss.com.my/en/product/lotuss-chips-110g-sour-cream onion-74700308
74696009 LOTUSS BLACK RAISIN 250G POUCH 7.99MYR https://www.lotuss.com.my/en/product/lotuss-black-raisin-250g-pouch-74696009
70856206 NESTUM 3-IN-1 CEREAL ORIGINAL 15X28G 9.99MYR https://www.lotuss.com.my/en/product/nestum-3-in-1-cereal-original-15x28g-70856206
74377485 SWISS LEF SWISS ROLL STRAWBERRY 6S 20G 2.79MYR https://www.lotuss.com.my/en/product/swiss-lef-swiss-roll-strawberry-6s-20g-74377485
74283677 COCON FRUIT CUBE PUDDING MANGO 118GX6 8.99MYR https://www.lotuss.com.my/en/product/cocon-fruit-cube-pudding-mango-118gx6-74283677
74547321 LE ORG EDAMAME FETTUCCINE 200G 12.99MYR https://www.lotuss.com.my/en/product/le-org-edamame-fettuccine-200g-74547321
74689584 LOTUSS BARLI 300G 2.49MYR https://www.lotuss.com.my/en/product/lotuss-barli-300g-74689584
120987 JACOBS WEETAMEAL 700G 13.5MYR https://www.lotuss.com.my/en/product/jacobs-weetameal-700g-120987
410675 KING CUP SARDINE 155G 4.19MYR https://www.lotuss.com.my/en/product/king-cup-sardine-155g-410675
410543 FAIZA EMAS SST 5% 10KG 24.29MYR https://www.lotuss.com.my/en/product/faiza-emas-sst-5-10kg-410543
3143996 YEOS CURRY CHICKEN 145G 4.95MYR https://www.lotuss.com.my/en/product/yeos-curry-chicken-145g-3143996
70323232 ARM / HAMMER PURE BAKING SODA 454G 7.29MYR https://www.lotuss.com.my/en/product/arm-&-hammer-pure-baking-soda-454g-70323232
72866039 B CROCKER PANCKE MIX BUTTERMILK 430G 9.99MYR https://www.lotuss.com.my/en/product/b-crocker-pancke-mix-buttermilk-430g-72866039
71428852 AH FOOK HERBS   SPICES MIXED 35G 5.99MYR https://www.lotuss.com.my/en/product/ah-fook-klang-herbs-&-spices-mixed-35g-71428852
73925624 LE CHILLI POWDER 60G 9.99MYR https://www.lotuss.com.my/en/product/le-chilli-powder-60g-73925624
72318899 CAMERON GARDEN BUTTON MUSHROOM CAN 425G 3.4MYR https://www.lotuss.com.my/en/product/cameron-garden-button-mushroom-can-425g-72318899
73785075 CED ORGANIC MOLASSES SUGAR 850G 13.99MYR https://www.lotuss.com.my/en/product/ced-organic-molasses-sugar-850g-73785075
74690175 LOTUSS COARSE SUGAR 1KG 2.85MYR https://www.lotuss.com.my/en/product/lotuss-coarse-sugar-1kg-74690175
74689649 LOTUSS KAYU MANIS 50G 3.79MYR https://www.lotuss.com.my/en/product/lotuss-kayu-manis-50g-74689649
74682237 LOTUSS SPAGHETTI 400G 2.49MYR https://www.lotuss.com.my/en/product/lotuss-spaghetti-400g-74682237
2544776 CAMPBELL CREAM OF MUSHROOM 3X21.1G 4.75MYR https://www.lotuss.com.my/en/product/campbell-cream-of-mushroom-3x21-1g-2544776
243183 COCON YOGO ICE COMBO PCK 10X45ML 5.09MYR https://www.lotuss.com.my/en/product/cocon-yogo-ice-combo-pck-10x45ml-243183
74738887 OHALO ORG COCONUT ALMOND SPREAD 135G 15.59MYR https://www.lotuss.com.my/en/product/ohalo-org-coconut-almond-spread-135g-74738887
2609347 JASMINE BEEHOON 400G 2.65MYR https://www.lotuss.com.my/en/product/jasmine-beehoon-400g-2609347
72486384 MUNCHYS OAT KRUNCH NUTTY CHOCOLATE 390G 7.99MYR https://www.lotuss.com.my/en/product/munchys-oat-krunch-nutty-chocolate-390g-72486384
71138722 POCKY STRAWBERRY FLAVOUR 38G 2.95MYR https://www.lotuss.com.my/en/product/pocky-strawberry-flavour-38g-71138722
74706527 LOTUSS MURUKU 200G PAKODA 3.99MYR https://www.lotuss.com.my/en/product/lotuss-muruku-200g-pakoda-74706527
74690426 LOTUSS CENDAWAN CINA 45G 5.49MYR https://www.lotuss.com.my/en/product/lotuss-cendawan-cina-45g-74690426
74557386 SAJI SANTAN KELAPA 200ML 2.9MYR https://www.lotuss.com.my/en/product/saji-santan-kelapa-200ml-74557386
74554093 LE COCONUT SYRUP 270G 13.99MYR https://www.lotuss.com.my/en/product/le-coconut-syrup-270g-74554093
74698869 LOTUSS HONEY 1KG 12.99MYR https://www.lotuss.com.my/en/product/lotuss-honey-1kg-74698869
4115252 TAO KAE NOI BIG BANG CLASSIC 50G 10.19MYR https://www.lotuss.com.my/en/product/tao-kae-noi-big-bang-classic-50g-4115252
71677100 MAMEE MONSTER  HOT SPICY 8X25G 3MYR https://www.lotuss.com.my/en/product/mamee-monster-family-pack-h s-8x25g-71677100
3806049 PENGUIN LYCHEE 567G 5.49MYR https://www.lotuss.com.my/en/product/penguin-lychee-567g-3806049
74698966 Lotus's Hot & Spicy Flavoured Potato Chips 70g 2.19MYR https://www.lotuss.com.my/en/product/lotuss-potato-chips-f-cut-70g-hot-spicy-74698966
74698435 LOTUSS CHICKEN STOCK CUBE 6X11G 2.89MYR https://www.lotuss.com.my/en/product/lotuss-chicken-stock-cube-6x11g-74698435
1041584 TESCO PITTED BLACK OLIVES 330G(D/W 163G) 6.21MYR https://www.lotuss.com.my/en/product/tesco-pitted-black-olives-330g-d-w-163g-1041584
378860 STAR BRAND FLAVOUR VANILLA 25ML 2.39MYR https://www.lotuss.com.my/en/product/star-brand-flavour-vanilla-25ml-378860
71002162 JIF PEANUT BUTTER - CRUNCHY 16OZ 16.29MYR https://www.lotuss.com.my/en/product/jif-peanut-butter-crunchy-16oz-71002162
3245926 YUPI STRAWBERRY KISS 120G 4.15MYR https://www.lotuss.com.my/en/product/yupi-strawberry-kiss-120g-3245926
2959399 SMUCKERS SUGAR FREE CONCORDE GRAPE 361G 11.59MYR https://www.lotuss.com.my/en/product/smuckers-sugar-free-concorde-grape-361g-2959399
117536 ADABI PERENCAH SUP SIAM 40G 1.35MYR https://www.lotuss.com.my/en/product/adabi-thai-soup-paste-40g-117536
518905 KOKO JELLY NUT 380G 7.59MYR https://www.lotuss.com.my/en/product/koko-jelly-nut-380g-518905
73973319 CFO RAISIN PROMO PACK 300GX2 25.9MYR https://www.lotuss.com.my/en/product/cfo-raisin-promo-pack-300gx2-73973319
74697234 LOTUSS CHEESE FLAVOUR RINGS 60G 1.09MYR https://www.lotuss.com.my/en/product/lotuss-cheese-flavour-rings-60g-74697234
74697056 LOTUSS SANDWICH SPREAD WITH TUNA 185G 4.99MYR https://www.lotuss.com.my/en/product/lotuss-sandwich-spread-with-tuna-185g-74697056
74706551 LOTUSS MURUKU 200G CAMPURAN 3.99MYR https://www.lotuss.com.my/en/product/lotuss-muruku-200g-campuran-74706551
3341828 MUNCHYS OATKRUNCH S/BERRY B/CURRANT 390G 7.99MYR https://www.lotuss.com.my/en/product/munchys-oatkrunch-s-berry-b-currant-390g-3341828
169080 KNIFE COOKING OIL 5KG 44.95MYR https://www.lotuss.com.my/en/product/knife-cooking-oil-5kg-169080
125946 NESTLE HONEY STARS ECONOPACK 500G 14.99MYR https://www.lotuss.com.my/en/product/nestle-honey-stars-econopack-500g-125946

Page link：https//www.codepudding.com/Mobile/480574.html

Prev:How to extract all attributes value inside nested android tag or how to extract the value/string ins
Next:Scrape only Tweets with Emojis in R
  • Related

About Us:  Contact Us      Terms of Service       Privacy Policy

Copyright © 2010-2023，Powered By CodePudding