How to scrape images from sliders/wrappers/carousels-CodePudding

I am trying to download the product pictures of bikes. On websites where the product pictures are in sliders/wrappers/carousels etc., I always run into problems. E.g. on this page: https://yeticycles.com/bikes/160e. BeautifulSoup simply does not find the div. In general, I am only interested in the first picture that is visible. Like in this screenshot.

I am trying it with the following code:

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.PhantomJS(executable_path='C:\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe')
driver.get('https://yeticycles.com/bikes/160e')
html = driver.page_source
soup = BeautifulSoup(html, 'lxml')
print(soup.find("div", {"class": 'base-picture bike-hero-image'}).find("img").get('src'))

CodePudding user response：

You can use just request for this

import requests
import json


def get_images(bike_name):
    url = f'https://bigcommerce-middleware-dot-yeticycles-prod.appspot.com/api/stores/5zdahzyh66/v1/products?keyword={bike_name}&include=images'
    response = requests.get(url)
    for data in json.loads(response.text)['data']:
        for image in data['images']:
            print(image['product_id'], image['description'], image['url_zoom'])


get_images('sb165')

For example i search sb165

Output:

1471 SB165 GEN1 PROTECTOR KIT 2020-CURRENT https://cdn11.bigcommerce.com/s-5zdahzyh66/products/1471/images/94573/9efc979cace7e423c8246633163938a4b96907c021e95bd59641647e66343488__29585.1648574414.1280.1280.jpg?c=2
1523 SB165 ALLOY LINK W/BEARING 2019-CURRENT https://cdn11.bigcommerce.com/s-5zdahzyh66/products/1523/images/94572/45540a43573afbbc14ad766479ac361456e1ffcfef2ad3bbc44118747e1742e6__69850.1648574408.1280.1280.jpg?c=2
2162 SB165 C2 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2162/images/95098/d03138a90da57ef1474c624ff11520ff4fe649c0b2c8245844953668489f346a__08353.1648582656.1280.1280.jpg?c=2
2162 SB165 C2 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2162/images/95099/48e119af933aae5c42ae9d600ff13b37183211fa8acf7a40fb0caf3f69dd5d48__83807.1648582656.1280.1280.jpg?c=2
2162 SB165 C2 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2162/images/95097/dd80792f55129dc0ee51d4d2a390fabd2be3f6fb1c6e0cbbb55f9350eebe2a2d__36945.1648582655.1280.1280.jpg?c=2
2163 SB165 Frame Only 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2163/images/95100/736752a4835ccaa30723bb1baf974700e95d896ca98dfda9935f6db20f4dbb5e__90861.1650389017.1280.1280.jpg?c=2
2163 SB165 Frame Only 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2163/images/95102/da67e9d1fae7418bc7f439867be65fd46a356bcb36a7466e91641b3b97eadfef__55929.1650389017.1280.1280.jpg?c=2
2163 SB165 Frame Only 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2163/images/95101/7372ecd1bf275438be866285381ddc6e8a82a8035a5a7b11eee918b217e76c53__80749.1650389017.1280.1280.jpg?c=2
2164 SB165 T1 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2164/images/95103/0c55ce5ae31ff388f99b37ef5c66cb4ebd320197d3a90b3b0efe2e5c2de59cfc__30942.1648582831.1280.1280.jpg?c=2
2164 SB165 T1 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2164/images/95104/038096dd4183b259b684bed7f1241188e945a6d417d4108b9132341909940a6d__32149.1648582832.1280.1280.jpg?c=2
2164 SB165 T1 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2164/images/95105/bee87286d60ed0153973a4427af7c3596cb02d04adc98a63794cfa88c8533109__61456.1648582832.1280.1280.jpg?c=2
2165 SB165 T2 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2165/images/95106/ec4ed7f67bd2dd7ca385b7395d34ce7bb045c2be2e0ac8e3c22302b769f7e261__93297.1648582969.1280.1280.jpg?c=2
2165 SB165 T2 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2165/images/95107/2861f0bee02770a7d0cd12589d65cc9a78a021b4572eb0f8102232905e9cfe0a__56123.1648582969.1280.1280.jpg?c=2
2165 SB165 T2 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2165/images/95108/a6b9d3ae08745bd4816883eef337652d44eb62801564e0843232f7a9bb4ce850__30020.1648582970.1280.1280.jpg?c=2
2167 SB165 T2 Float X2 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2167/images/95113/ab68411c29a68b2a1703175ac22b39fd8d27e568ebb3ca73789cc7562993dd36__70793.1648583249.1280.1280.jpg?c=2
2167 SB165 T2 Float X2 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2167/images/95114/328f69577ffaf6cc5589ae9d93dd16e59778759a01e78ee46b9fc525a95e5733__60402.1648583250.1280.1280.jpg?c=2
2167 SB165 T2 Float X2 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2167/images/95112/2c35b0d46d15785a7c47b88c60b8387c9176fb2368cf05b4b36c8f42e6602d80__67592.1648583249.1280.1280.jpg?c=2
2168 SB165 C1 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2168/images/95094/5ea527ff32e1bdd26f5fec4e3c0282cf381a2e71367031cde58dde0711684ed0__54493.1648582564.1280.1280.jpg?c=2
2168 SB165 C1 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2168/images/95096/12d2046fcdb65c2b70137c1360f9b7cddcec817192d6a5e9cde9c33f96c91a3d__66221.1648582565.1280.1280.jpg?c=2
2168 SB165 C1 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2168/images/95095/3818ebe31cbd7f5640e61e527f9d4de818440e3a1c6cebe7929a20934a284419__55721.1648582565.1280.1280.jpg?c=2
2213 SB165 T3 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2213/images/95110/b06f6a6bec9d043bfcc95fcd3eeaf96f497ff2bdd06b913c24f730c7077e49b1__77595.1648583108.1280.1280.jpg?c=2
2213 SB165 T3 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2213/images/95111/e071794aaef3e54ce5c1da0d1371051c3deb1d3f07f3ab4b93532589cc6b9296__80594.1648583109.1280.1280.jpg?c=2
2213 SB165 T3 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2213/images/95109/4f5312cbfaaa9744e2f960176cd6dbf9f60ecc3572e1cac0ddcfabd8f2aca762__05931.1648583108.1280.1280.jpg?c=2
2253 SB165 C1.5 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2253/images/95207/5ea527ff32e1bdd26f5fec4e3c0282cf381a2e71367031cde58dde0711684ed0__40726.1648587902.1280.1280.jpg?c=2
2253 SB165 C1.5 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2253/images/95209/12d2046fcdb65c2b70137c1360f9b7cddcec817192d6a5e9cde9c33f96c91a3d__43062.1648587903.1280.1280.jpg?c=2
2253 SB165 C1.5 2022 https://cdn11.bigcommerce.com/s-5zdahzyh66/products/2253/images/95208/3818ebe31cbd7f5640e61e527f9d4de818440e3a1c6cebe7929a20934a284419__96181.1648587903.1280.1280.jpg?c=2

CodePudding user response：

You can use requests_html to render page and scroll down

from requests_html import HTMLSession
session = HTMLSession()
url = "https://yeticycles.com/bikes/160e"
r = session.get(url)
r.html.render(sleep=1, scrolldown=10)
x = r.html.find(".base-picture.bike-hero-image picture source")
for i in x:
    print(i.attrs["srcset"])

Output

https://cdn11.bigcommerce.com/s-5zdahzyh66/images/stencil/2880w/attribute_rule_images/592061_source_1648586055.jpg
https://cdn11.bigcommerce.com/s-5zdahzyh66/images/stencil/2880w/attribute_rule_images/592060_source_1648586050.jpg

to install library use pip install requests-html