I'm trying to scrape some information from the following webpage https://smws.com/all-whisky?sort=pricedesc&page=1&per-page=128
The information I'm trying to scrape is the name of each item plus the product code (not quite there yet with the below code). For example, for the first item this would be "Goodnight and Joy" & "53.389". I then wish to iterate this over the whole webpage. My code is as follows:
import requests
import time
import smtplib
from email.message import EmailMessage
import hashlib
from urllib.request import urlopen
from datetime import datetime
from bs4 import BeautifulSoup
from selenium import webdriver
url = 'https://smws.com/all-whisky?sort=pricedesc&page=1&per-page=128'
soup = BeautifulSoup(urlopen(url).read(),features="html.parser")
test = soup.find_all('a.href')
print(test)
However this simply returns
[]
Please could someone help me to return the information returned in a.href and potentially iterated for the entire page.
Thank you!
CodePudding user response：
please see the code below, its working good--
url = 'https://smws.com/all-whisky?sort=pricedesc&page=1&per-page=128'
r = requests.get(url)
soup = BeautifulSoup(r.content,features="html.parser")
for link in soup.find_all('a'):
print(link.get('href'))
CodePudding user response：
The correct way to get the link [or href attributes] from every a tag is not find_all('a.href') as you have used it. You can instead do something like
test = [s for s in soup.find_all('a') if s.get('href')]
#for just the links:
#test = [s.get('href') for s in soup.find_all('a') if s.get('href')]
and get the a tags that have href and are available in the html you fetched.
I then printed with
print(f'{len(test)} results:\n')
for t in test: print(f'[{t.get_text(strip=True)}]({t.get("href")})')
to view both the text and the link in each tag.
However that will still not get you the data you mentioned wanting
the name of each item plus the product code
because that has not been loaded in the html you fetched (you can check with a print(soup.text)...and also, if you try print(soup.find('ul', {"id":"product-listing-container"}).prettify()) you will see that ul#product-listing-container is filled with "skeleton products" instead of having the article tag filled with attributes as on the page after products have loaded).
But I also noticed you have selenium in your imports, so it's easily resolved with that (you'll just have to have it confirm you're an adult and then wait for products to load and then use bs4 or just directly call selenium's find_elements...)
CodePudding user response：
Information in that page is being hydrated via XHR calls, by javascript. If you want to reach that information with Requests, you need to scrape the actual API endpoints. Here is one way to do it:
import requests
import pandas as pd
pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', None)
r = requests.get('https://api.smws.com/api/v1/bottles?store_id=uk&parent_id=61&page=1&sortBy=pricedesc&minPrice=0&maxPrice=0&perPage=128')
df = pd.json_normalize(r.json()['items'])
print(df)
Result printed in terminal:
id sku name url cask_no region age abv cask_type spirit_type list_description price stock release_date distilleddate image product_categories categories
0 3072 004343GB0700631 Analytical allure /analytical-allure/ 4.343 Highland 31.0 53.6% 2nd fill ex-madeira hogshead Malt Whisky None 695.00 24 2022-10-19 13:00:00 1/5/1991 https://cdn11.bigcommerce.com/s-vagfena5nz/products/3072/images/4821/4.343-web__47052.1666104996.386.513.png?c=1 All Whisky/Flavour Profiles, All Whisky/Whisky by Region, All Whisky/Whisky by Age, All Whisky, All Whisky/Cask Type, All Whisky/Whisky/Outturn, All Whisky/Whisky, Whisky/Outturn/Mid-Month Releases, All Whisky/Whisky by Region/Highland, All Whisky/Whisky/Vaults Collection, All Whisky/Flavour Profiles/Lightly Peated, All Whisky/Whisky by Age/21 Years, All Whisky/Cask Type/Other Fortified Wines & Wine Casks, All Whisky/Whisky/Spirit of the Season [All Whisky/Flavour Profiles, All Whisky/Whisky by Region, All Whisky/Whisky by Age, All Whisky, All Whisky/Cask Type, All Whisky/Whisky/Outturn, All Whisky/Whisky, Whisky/Outturn/Mid-Month Releases, All Whisky/Whisky by Region/Highland, All Whisky/Whisky/Vaults Collection, All Whisky/Flavour Profiles/Lightly Peated, All Whisky/Whisky by Age/21 Years, All Whisky/Cask Type/Other Fortified Wines & Wine Casks, All Whisky/Whisky/Spirit of the Season]
1 2826 053389GB0700630 Goodnight and joy /goodnight-and-joy/ 53.389 Islay 30.0 55.5% refill ex-sherry butt Malt Whisky None 495.00 15 2022-07-25 08:00:00 21/6/1991 https://cdn11.bigcommerce.com/s-vagfena5nz/products/2826/images/4416/53.389-web__29542.1658745823.386.513.png?c=1 All Whisky/Whisky by Age/21 Years, All Whisky/Flavour Profiles, All Whisky/Flavour Profiles/Peated, All Whisky/Whisky by Region, All Whisky/Whisky by Region/Islay, All Whisky/Whisky by Age, All Whisky, All Whisky/Single Cask Whisky, All Whisky/Whisky/Vaults Collection, All Whisky/Cask Type/Sherry Casks, No Menu Link/External Tastings, No Menu Link/External Tastings/Sept22 Tasting Bottles, All Whisky/Cask Type, All Whisky/Whisky/Last Drops [All Whisky/Whisky by Age/21 Years, All Whisky/Flavour Profiles, All Whisky/Flavour Profiles/Peated, All Whisky/Whisky by Region, All Whisky/Whisky by Region/Islay, All Whisky/Whisky by Age, All Whisky, All Whisky/Single Cask Whisky, All Whisky/Whisky/Vaults Collection, All Whisky/Cask Type/Sherry Casks, No Menu Link/External Tastings, No Menu Link/External Tastings/Sept22 Tasting Bottles, All Whisky/Cask Type, All Whisky/Whisky/Last Drops]
2 1124 095039GB0700638 Indian summer in a Japanese garden /indian-summer-in-a-japanese-garden/ 95.39 Speyside 38.0 53.0% Refill ex-sherry butt Malt Whisky None 435.00 35 2021-06-17 13:00:00 15/1/1982 https://cdn11.bigcommerce.com/s-vagfena5nz/products/1124/images/1770/95.39-VC-box-web__23227.1623844189.386.513.png?c=1 All Whisky/Whisky by Region/Speyside, All Whisky/Whisky by Age/21 Years, All Whisky/Flavour Profiles, All Whisky/Flavour Profiles/Old & Dignified, All Whisky/Whisky by Region, All Whisky/Whisky by Age, All Whisky, All Whisky/Single Cask Whisky, All Whisky/Whisky/Vaults Collection, All Whisky/Cask Type/Sherry Casks, All Whisky/Cask Type [All Whisky/Whisky by Region/Speyside, All Whisky/Whisky by Age/21 Years, All Whisky/Flavour Profiles, All Whisky/Flavour Profiles/Old & Dignified, All Whisky/Whisky by Region, All Whisky/Whisky by Age, All Whisky, All Whisky/Single Cask Whisky, All Whisky/Whisky/Vaults Collection, All Whisky/Cask Type/Sherry Casks, All Whisky/Cask Type]
3 2111 012064GB0700631 Oh-so-orange /oh-so-orange/ 12.64 Speyside 31.0 43.4% 2nd fill ex-bourbon barrel Malt Whisky None 434.65 11 2022-02-11 09:00:00 7/5/1990 https://cdn11.bigcommerce.com/s-vagfena5nz/products/2111/images/3102/12.64-VC-box-hires__68013.1657010779.386.513.png?c=1 All Whisky/Whisky by Region/Speyside, All Whisky/Whisky by Age/21 Years, Categories Removed, All Whisky/Flavour Profiles, All Whisky/Flavour Profiles/Sweet Fruity & Mellow, All Whisky/Whisky by Region, All Whisky/Whisky by Age, All Whisky, All Whisky/Single Cask Whisky, All Whisky/Whisky/Vaults Collection, All Whisky/Cask Type/Bourbon Casks, All Whisky/Cask Type, Last Drops, All Whisky/Whisky/Last Drops [All Whisky/Whisky by Region/Speyside, All Whisky/Whisky by Age/21 Years, Categories Removed, All Whisky/Flavour Profiles, All Whisky/Flavour Profiles/Sweet Fruity & Mellow, All Whisky/Whisky by Region, All Whisky/Whisky by Age, All Whisky, All Whisky/Single Cask Whisky, All Whisky/Whisky/Vaults Collection, All Whisky/Cask Type/Bourbon Casks, All Whisky/Cask Type, Last Drops, All Whisky/Whisky/Last Drops]
4 2897 012062GB0700631 Heaven-sent /heaven-sent/ 12.62 Speyside 31.0 57.9% 1st fill ex-Pedro Ximenes butt Malt Whisky None 385.00 5 2022-08-22 11:00:00 7/9/1989 https://cdn11.bigcommerce.com/s-vagfena5nz/products/2897/images/4563/12.62-VC-box-web_1__60925.1660742340.386.513.png?c=1 All Whisky/Flavour Profiles, All Whisky/Flavour Profiles/Deep Rich & Dried Fruits, All Whisky/Whisky by Region, All Whisky/Whisky by Age, All Whisky, All Whisky/Whisky/Vaults Collection, All Whisky/Whisky by Region/Speyside, All Whisky/Whisky by Age/21 Years, All Whisky/Cask Type/Sherry Casks, All Whisky/Cask Type, All Whisky/Whisky/Last Drops [All Whisky/Flavour Profiles, All Whisky/Flavour Profiles/Deep Rich & Dried Fruits, All Whisky/Whisky by Region, All Whisky/Whisky by Age, All Whisky, All Whisky/Whisky/Vaults Collection, All Whisky/Whisky by Region/Speyside, All Whisky/Whisky by Age/21 Years, All Whisky/Cask Type/Sherry Casks, All Whisky/Cask Type, All Whisky/Whisky/Last Drops]
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
94 2898 006058GB0700608 Red swirl lollipop in a sidecar /red-swirl-lollipop-in-a-sidecar/ 6.58 Speyside 8.0 65.5% 2nd fill ex-bourbon barrel Malt Whisky None 49.80 14 2022-08-19 08:00:00 18/9/2013 https://cdn11.bigcommerce.com/s-vagfena5nz/products/2898/images/4565/6.58-web__01010.1660745386.386.513.png?c=1 All Whisky/Flavour Profiles, All Whisky/Flavour Profiles/Spicy & Sweet, All Whisky/Whisky by Region, All Whisky/Whisky by Age, All Whisky, All Whisky/Single Cask Whisky, All Whisky/Whisky by Age/Under 12 Years, All Whisky/Whisky by Region/Speyside, All Whisky/Whisky/Whisky Best Buys, All Whisky/Cask Type/Bourbon Casks, All Whisky/Cask Type [All Whisky/Flavour Profiles, All Whisky/Flavour Profiles/Spicy & Sweet, All Whisky/Whisky by Region, All Whisky/Whisky by Age, All Whisky, All Whisky/Single Cask Whisky, All Whisky/Whisky by Age/Under 12 Years, All Whisky/Whisky by Region/Speyside, All Whisky/Whisky/Whisky Best Buys, All Whisky/Cask Type/Bourbon Casks, All Whisky/Cask Type]
95 3025 GN6002GB0700601 Old timey sippin' gin! /old-timey-sippin-gin/ GN6.2 Edinburgh 1.0 49.3% 1st fill ex-bourbon barrel Malt Whisky None 45.75 48 2022-10-07 08:00:00 10/9/2020 https://cdn11.bigcommerce.com/s-vagfena5nz/products/3025/images/4741/GN6.2-web__93878.1664901852.386.513.png?c=1 All Whisky/Whisky by Age, All Whisky, All Whisky/Single Cask Whisky, All Whisky/Whisky by Age/Under 12 Years, All Whisky/Cask Type, All Whisky/Cask Type/Bourbon Casks, All Whisky/Spirits, All Whisky/Spirits/Gin, All Whisky/Whisky, All Whisky/Whisky/Outturn [All Whisky/Whisky by Age, All Whisky, All Whisky/Single Cask Whisky, All Whisky/Whisky by Age/Under 12 Years, All Whisky/Cask Type, All Whisky/Cask Type/Bourbon Casks, All Whisky/Spirits, All Whisky/Spirits/Gin, All Whisky/Whisky, All Whisky/Whisky/Outturn]
96 942 048111GB0700607 Baking in the potting shed /baking-in-the-potting-shed/ 48.111 Speyside 9.0 61.8% 1st fill ex-bourbon barrel None None 45.60 24 2022-10-07 08:00:00 7/11/2011 https://cdn11.bigcommerce.com/s-vagfena5nz/products/942/images/1391/48.111-bc__44272.1617989677.386.513.png?c=1 All Whisky/Whisky by Age/Under 12 Years, All Whisky/Whisky by Region/Speyside, All Whisky/Flavour Profiles/Spicy & Sweet, All Whisky, All Whisky/Single Cask Whisky, All Whisky/Cask Type/Bourbon Casks, All Whisky/Cask Type, All Whisky/Flavour Profiles, All Whisky/Whisky by Region, All Whisky/Whisky by Age, All Whisky/Whisky, All Whisky/Whisky/Outturn [All Whisky/Whisky by Age/Under 12 Years, All Whisky/Whisky by Region/Speyside, All Whisky/Flavour Profiles/Spicy & Sweet, All Whisky, All Whisky/Single Cask Whisky, All Whisky/Cask Type/Bourbon Casks, All Whisky/Cask Type, All Whisky/Flavour Profiles, All Whisky/Whisky by Region, All Whisky/Whisky by Age, All Whisky/Whisky, All Whisky/Whisky/Outturn]
97 345 SM0001GX0700607 Lemon odyssey /lemon-odyssey SM0.1 Blended Malt 7.0 50.0% 1st & 2nd fill ex-bourbon barrels Malt Whisky None 45.00 50 None 11/6/2012 https://cdn11.bigcommerce.com/s-vagfena5nz/products/345/images/3979/Lemon-Odyssey-web__34434.1651854289.386.513.png?c=1 All Whisky/Whisky by Age/Under 12 Years, All Whisky/Whisky/Whisky Highlights, All Whisky/Flavour Profiles, All Whisky/Whisky by Region, All Whisky/Whisky by Region/International, All Whisky/Whisky by Age, All Whisky, All Whisky/Single Cask Whisky [All Whisky/Whisky by Age/Under 12 Years, All Whisky/Whisky/Whisky Highlights, All Whisky/Flavour Profiles, All Whisky/Whisky by Region, All Whisky/Whisky by Region/International, All Whisky/Whisky by Age, All Whisky, All Whisky/Single Cask Whisky]
98 2904 DISC-VMT-GATH22-GB Christmas Selection Box /Gathering-Discovery-Pack-2022 None None NaN None None None Pick up a perfect present for the whisky lover in your life or get in the seasonal spirit yourself – either with a spot of self-exploration or by tuning into our online tasting, where our ambassador will guide you through the selection. 30.00 107 2022-08-22 08:00 None https://cdn11.bigcommerce.com/s-vagfena5nz/products/2904/images/4578/TP_May_2020_Festival_01s__44827.1662024217.386.513.jpg?c=1 All Whisky/Whisky/Whisky Best Buys, All Whisky/Whisky/Outturn, All Whisky/Flavour Profiles, All Whisky/Whisky by Region, All Whisky/Whisky by Age, All Whisky, All Whisky/Whisky/Cases & Tasting Packs, All Whisky/Whisky/Whisky Gifts for all Occasions, All Whisky/Whisky/Spirit of the Season [All Whisky/Whisky/Whisky Best Buys, All Whisky/Whisky/Outturn, All Whisky/Flavour Profiles, All Whisky/Whisky by Region, All Whisky/Whisky by Age, All Whisky, All Whisky/Whisky/Cases & Tasting Packs, All Whisky/Whisky/Whisky Gifts for all Occasions, All Whisky/Whisky/Spirit of the Season]
99 rows × 18 columns
See pandas documentation here: https://pandas.pydata.org/docs/
Also, Requests documentation: https://requests.readthedocs.io/en/latest/