Home > Back-end >  How do i declare data-test-id attribute inside web scraping script
How do i declare data-test-id attribute inside web scraping script

Time:04-14

I am trying to scrape images from the toronto star news website (https://thestar.com), I want to scrape all the images with a data-test-id attribute of "medicard-image".

Here's the code i am running to preview the images i have scraped:

ht_r = requests.get("https://www.thestar.com/")
ht_soup = BeautifulSoup(ht_r.content, 'html.parser')
ht_images = ht_soup.findAll('img', attr={"data-test-id": "medicard-image"})
ht_images = ht_images[0:14]

However whenever i try to print ht_images , i get an output of [].

What am i doing wrong?

CodePudding user response:

The images you see on the page are injected via JavaScript so BeautifulSoup doesn't see them. You can use this code to print all image URLs on page:

import re
import json
import requests


ht_r = requests.get("https://www.thestar.com/")
data = re.search(r"window\.__PRELOADED_STATE__ = (.*)", ht_r.text)
data = json.loads(data.group(1))

# uncomment to print all data:
# print(json.dumps(data, indent=4))

def find_images(d):
    if isinstance(d, dict):
        for k, v in d.items():
            if k == "image":
                yield v
            else:
                yield from find_images(v)
    if isinstance(d, list):
        for v in d:
            yield from find_images(v)


for img in find_images(data):
    if img["url"]:
        print("https://www.thestar.com"   img["url"])

Prints:

https://www.thestar.com/content/dam/thestar/politics/political-opinion/2022/04/12/justin-trudeau-has-an-untested-plan-to-create-jobs-will-canadians-have-the-patience-to-see-if-it-works/chrystia_freeland.jpg
https://www.thestar.com/content/dam/thestar/opinion/star-columnists/2022/04/12/blasting-kieran-moore-for-his-vacation-well-need-him-on-our-side-for-bigger-battles/kieran_moore.jpg
https://www.thestar.com/content/dam/thestar/opinion/star-columnists/2022/04/12/as-omicron-rises-are-things-falling-apart-are-we/thief_on_camera.jpg
https://www.thestar.com/content/dam/thestar/entertainment/opinion/2022/04/12/if-britney-spears-really-is-pregnant-its-time-to-get-off-social-media/britney_and_sam.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/toronto-police-chief-set-to-announce-arrests-in-daylight-murders-of-international-student-and-35-year-old-man/victims_2_.jpg
https://www.thestar.com/content/dam/thestar/politics/provincial/2022/04/12/doug-ford-draws-fire-for-comments-during-top-doctors-caribbean-vacation/doug_ford.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/just-15-per-cent-of-homeowners-hold-nearly-one-third-of-ontarios-housing-wealth-says-report/for_lease.jpg
https://www.thestar.com/content/dam/thestar/politics/federal/2022/04/12/canada-is-fighting-russias-invasion-of-ukraine-with-sanctions-but-are-they-actually-working/vladimir_putin.jpg
https://www.thestar.com/content/dam/thestar/uploads/2022/04/12/0799a6092c33428f8541cc8a6fa64c28-0799a6092c33428f8541cc8a6fa64c2-1.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/i-beg-you-mother-pleads-for-information-in-shocking-wasaga-beach-abduction-of-elnaz-hajtamiri/_1_elnaz.jpg
https://www.thestar.com/content/dam/thestar/politics/provincial/2022/04/12/ndp-says-it-will-review-policy-of-allowing-sitting-members-to-be-challenged-for-right-to-run-in-elections-andrea-horwath-says/horwath.jpg
https://www.thestar.com/content/dam/thestar/entertainment/2022/04/12/gilbert-gottfriend-has-died-at-age-67-heres-how-his-fellow-actors-and-comics-are-reacting-to-the-news/_37c33656f726494595f3491faf2d450c_37c33656f726494595f3491faf2d450.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/toronto-police-officer-faces-eight-charges-of-theft/toronto_police_cruiser.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/73-year-old-toronto-man-charged-with-kidnapping-at-royal-york-station/cruiser.jpg
https://www.thestar.com/content/dam/thestar/business/2022/04/12/to-mask-or-not-to-mask-without-ontario-backing-businesses-face-potential-legal-landmines/gil_2.jpg
https://www.thestar.com/content/dam/thestar/business/2022/04/12/this-will-remove-buyers-from-the-market-higher-interest-rates-on-wednesday-will-help-tame-frothy-housing-prices-experts-say/housing_sales.jpg
https://www.thestar.com/content/dam/thestar/news/canada/2022/04/12/coronavirus-covid-19-updates-toronto-canada-april-12/ptw_features_april_04.jpg
https://www.thestar.com/content/dam/thestar/news/canada/2022/04/11/pierre-poilievre-did-a-campaign-video-about-this-vancouver-house-heres-the-story-of-how-its-price-climbed-from-265000-to-49-million/house.jpg
https://www.thestar.com/content/dam/thestar/uploads/2022/02/28/headerimage-0-0-0.jpg
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/13-year-old-boy-granted-bail-in-shooting-death-of-toronto-boy-15/afwyouthjail02.jpg
https://www.thestar.com/content/dam/localcommunities/brampton_guardian/news/crime/2022/04/12/brampton-father-confronts-alleged-carjackers-trying-to-steal-suv-with-baby-in-back-seat/10605582_Bramptoncarjacking.JPG
https://www.thestar.com/content/dam/thestar/news/gta/2022/04/12/donated-toys-were-a-big-hit-with-kids-and-parents-at-this-toronto-playground-then-the-city-tossed-them/sr_cudmorecreekpark_01.jpg
https://www.thestar.com/content/dam/thestar/sports/leafs/2022/04/12/leafs-star-auston-matthews-is-outta-control-as-he-seeks-goal-no-60-against-sabres/matthews.jpg
https://www.thestar.com/content/dam/thestar/sports/raptors/analysis/2022/04/12/raptors-coach-nick-nurse-is-in-demand-for-a-reason-he-demands-success-and-his-bag-of-tricks-is-deep/nick_nurse.jpg
https://www.thestar.com/content/dam/thespec/news/hamilton-region/2022/04/12/dominican-republic-pivot-airlines-cocaine-bust/aatif_safdar.jpg
https://www.thestar.com/content/dam/thestar/uploads/2022/04/12/real-estate-sold-sign-13superportrait-0.jpg
https://www.thestar.com/content/dam/localcommunities/scarborough_mirror/News/gta/2022/04/12/toronto-zoo-aims-to-keep-its-new-and-expectant-moms-healthy-happy/10605993_DawnMihailovicwithHermanVancouverIslandMarmot.JPG

...and so on.
  • Related