I am trying to scrape images from the toronto star news website (https://thestar.com),
I want to scrape all the images with a data-test-id attribute of "medicard-image".
Here's the code i am running to preview the images i have scraped:
ht_r = requests.get("https://www.thestar.com/")
ht_soup = BeautifulSoup(ht_r.content, 'html.parser')
ht_images = ht_soup.findAll('img', attr={"data-test-id": "medicard-image"})
ht_images = ht_images[0:14]
However whenever i try to print ht_images , i get an output of [].
What am i doing wrong?
CodePudding user response：
The images you see on the page are injected via JavaScript so BeautifulSoup doesn't see them. You can use this code to print all image URLs on page:
import re
import json
import requests
ht_r = requests.get("https://www.thestar.com/")
data = re.search(r"window\.__PRELOADED_STATE__ = (.*)", ht_r.text)
data = json.loads(data.group(1))
# uncomment to print all data:
# print(json.dumps(data, indent=4))
def find_images(d):
if isinstance(d, dict):
for k, v in d.items():
if k == "image":
yield v
else:
yield from find_images(v)
if isinstance(d, list):
for v in d:
yield from find_images(v)
for img in find_images(data):
if img["url"]:
print("https://www.thestar.com" img["url"])
Prints:
https://www.thestar.com/content/dam/thestar/politics/political-opinion/2022/04/12/justin-trudeau-has-an-untested-plan-to-create-jobs-will-canadians-have-the-patience-to-see-if-it-works/chrystia_freeland.jpg
https://www.thestar.com/content/dam/thestar/opinion/star-columnists/2022/04/12/blasting-kieran-moore-for-his-vacation-well-need-him-on-our-side-for-bigger-battles/kieran_moore.jpg
https://www.thestar.com/content/dam/thestar/opinion/star-columnists/2022/04/12/as-omicron-rises-are-things-falling-apart-are-we/thief_on_camera.jpg
https://www.thestar.com/content/dam/thestar/entertainment/opinion/2022/04/12/if-britney-spears-really-is-pregnant-its-time-to-get-off-social-media/britney_and_sam.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/toronto-police-chief-set-to-announce-arrests-in-daylight-murders-of-international-student-and-35-year-old-man/victims_2_.jpg
https://www.thestar.com/content/dam/thestar/politics/provincial/2022/04/12/doug-ford-draws-fire-for-comments-during-top-doctors-caribbean-vacation/doug_ford.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/just-15-per-cent-of-homeowners-hold-nearly-one-third-of-ontarios-housing-wealth-says-report/for_lease.jpg
https://www.thestar.com/content/dam/thestar/politics/federal/2022/04/12/canada-is-fighting-russias-invasion-of-ukraine-with-sanctions-but-are-they-actually-working/vladimir_putin.jpg
https://www.thestar.com/content/dam/thestar/uploads/2022/04/12/0799a6092c33428f8541cc8a6fa64c28-0799a6092c33428f8541cc8a6fa64c2-1.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/i-beg-you-mother-pleads-for-information-in-shocking-wasaga-beach-abduction-of-elnaz-hajtamiri/_1_elnaz.jpg
https://www.thestar.com/content/dam/thestar/politics/provincial/2022/04/12/ndp-says-it-will-review-policy-of-allowing-sitting-members-to-be-challenged-for-right-to-run-in-elections-andrea-horwath-says/horwath.jpg
https://www.thestar.com/content/dam/thestar/entertainment/2022/04/12/gilbert-gottfriend-has-died-at-age-67-heres-how-his-fellow-actors-and-comics-are-reacting-to-the-news/_37c33656f726494595f3491faf2d450c_37c33656f726494595f3491faf2d450.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/toronto-police-officer-faces-eight-charges-of-theft/toronto_police_cruiser.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/73-year-old-toronto-man-charged-with-kidnapping-at-royal-york-station/cruiser.jpg
https://www.thestar.com/content/dam/thestar/business/2022/04/12/to-mask-or-not-to-mask-without-ontario-backing-businesses-face-potential-legal-landmines/gil_2.jpg
https://www.thestar.com/content/dam/thestar/business/2022/04/12/this-will-remove-buyers-from-the-market-higher-interest-rates-on-wednesday-will-help-tame-frothy-housing-prices-experts-say/housing_sales.jpg
https://www.thestar.com/content/dam/thestar/news/canada/2022/04/12/coronavirus-covid-19-updates-toronto-canada-april-12/ptw_features_april_04.jpg
https://www.thestar.com/content/dam/thestar/news/canada/2022/04/11/pierre-poilievre-did-a-campaign-video-about-this-vancouver-house-heres-the-story-of-how-its-price-climbed-from-265000-to-49-million/house.jpg
https://www.thestar.com/content/dam/thestar/uploads/2022/02/28/headerimage-0-0-0.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/13-year-old-boy-granted-bail-in-shooting-death-of-toronto-boy-15/afwyouthjail02.jpg
https://www.thestar.com/content/dam/localcommunities/brampton_guardian/news/crime/2022/04/12/brampton-father-confronts-alleged-carjackers-trying-to-steal-suv-with-baby-in-back-seat/10605582_Bramptoncarjacking.JPG
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/donated-toys-were-a-big-hit-with-kids-and-parents-at-this-toronto-playground-then-the-city-tossed-them/sr_cudmorecreekpark_01.jpg
https://www.thestar.com/content/dam/thestar/sports/leafs/2022/04/12/leafs-star-auston-matthews-is-outta-control-as-he-seeks-goal-no-60-against-sabres/matthews.jpg
https://www.thestar.com/content/dam/thestar/sports/raptors/analysis/2022/04/12/raptors-coach-nick-nurse-is-in-demand-for-a-reason-he-demands-success-and-his-bag-of-tricks-is-deep/nick_nurse.jpg
https://www.thestar.com/content/dam/thespec/news/hamilton-region/2022/04/12/dominican-republic-pivot-airlines-cocaine-bust/aatif_safdar.jpg
https://www.thestar.com/content/dam/thestar/uploads/2022/04/12/real-estate-sold-sign-13superportrait-0.jpg
https://www.thestar.com/content/dam/localcommunities/scarborough_mirror/News/gta/2022/04/12/toronto-zoo-aims-to-keep-its-new-and-expectant-moms-healthy-happy/10605993_DawnMihailovicwithHermanVancouverIslandMarmot.JPG
...and so on.