i'm trying to understand why my soup.find_all only returns 20 results, even though there should be more of those.

#Site
url = 'https://www.fairprice.com.sg/promotions'

#requests
headers = {'user-agent': 'my-agent/1.0.1'}
r = requests.get(url, headers=headers)

#bs4
soup = BeautifulSoup(r.text,'html.parser')
mydivs = soup.find_all("div", {"class": "sc-1plwklf-10 jNMebL"})

len(mydivs)
Output: 40

Searched through the sites and have realised it may be due to the page loading the products after scrolling down. Is there a workaround for this? Appreciate the help.

CodePudding user response：

You are getting 20 items as output because rest of data are dynamically populated by JavaScript as GET method from API.

import requests
import pandas as pd

experiments = ('searchVariant-B,timerVariant-Z,inlineBanner-A,'
               'substitutionBSVariant-A,gv-A,SPI-Z,SNLI-B,SC-B,'
               'shelflife-C,A')

params = dict(experiments=experiments,
              includeTagDetails='true',
              orderType='DELIVERY',
              pageType='promotion',
              storeId=165,
              )

url = 'https://website-api.omni.fairprice.com.sg/api/product/v2'

result=[]
for params['page'] in range(1, 6):
    data = requests.get(url, params).json()
    for item in data['data']['product']:
        name = item['name']
        result.append({"Product name": name})
df = pd.DataFrame(result)
print(df)

Output:

                                    Product name
0                               Pasar Fresh Eggs
1                     Meiji Fresh Milk - Regular
2     FairPrice DeluxSoft Bathroom Tissue (3ply)
3                              Fresh Blueberries
4         UFC Refresh 100% Natural Coconut Water
..                                           ...
95                Magnum Mini Ice Cream - Almond
96  Clorox Toilet Bowl Cleaner Tablets - Tru Blu
97     Kleenex Facial Tissue Box - Floral (3ply)
98                    Fairprice Scouring Sponge 
99        Prego Pasta Sauce - Traditional Tomato

[100 rows x 1 columns]