I am trying to scrape images in 'solid' tab under finishing color options within "https://www.novitecgroup.com/en/brands/ferrari/roma/#wheels". For that am getting outer html and then trimming it out to get required image link. I have tried using code lines, they worked for transparent but here(for solid) they don't seem to return any thing. Even tried manual clicking on that tab but nothing happens, rather code times out eventually.
featherlight = WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.rims__solid.active")))
time.sleep(5)
for k in range(1,25):
path_link_DP_Solid = "//div[@class='colorCols']/div[" str(k) "]/p[1]/img[1]"
Collected_link_DP_Solid = featherlight.find_element(By.XPATH, path_link_DP_Solid).get_attribute('outerHTML')
CodePudding user response：
No need for Selenium here. I originally had it iterate through the 3 types, but noticed that the solid rim colors are the same for all 3. So just grabbed the first. I'm making that assumption for the transparent:
import requests
from bs4 import BeautifulSoup
import re
imageLinks = {}
url = 'https://www.novitecgroup.com/en/brands/ferrari/roma/tuningpart/1536'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
for each in ['transparent', 'solid']:
imageLinks[each] = []
rims = soup.find('div', {'class':re.compile(f"^rims__{each}")})
images = rims.find_all('img')
for img in images:
imageLinks[each].append(img['src'])
Output:
print(imageLinks)
{'transparent': ['/vossen-felgen/cap.jpg', '/vossen-felgen/angled/1-Midnight-Smoke-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/1-Midnight-Smoke-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/2-Dark-Smoke-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/2-Dark-Smoke-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/3--Light-Smoke-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/3--Light-Smoke-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/4-Gloss-Clear-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/4-Gloss-Clear-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/5-Stealth-Grey-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/5-Stealth-Grey-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/7-Space-Grey-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/7-Space-Grey-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/6-Platinum-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/6-Platinum-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/8-Matte-Clear-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/8-Matte-Clear-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/9-Patina-Gold-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/9-Patina-Gold-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/10-Honeycomb-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/10-Honeycomb-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/11-Champagne-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/11-Champagne-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/12-Vintage-Ros-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/12-Vintage-Ros-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/13-Amber-Tone-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/13-Amber-Tone-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/14-Copper-Penny-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/14-Copper-Penny-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/15-Bronze-Brickell-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/15-Bronze-Brickell-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/16-Imperial-Gold-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/16-Imperial-Gold-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/17-Merlot-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/17-Merlot-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/18-Vossen-Red-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/18-Vossen-Red-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/19-Scarlet-Red-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/19-Scarlet-Red-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/20-Flamingo-Pink-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/20-Flamingo-Pink-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/21-Deco-Teal-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/21-Deco-Teal-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/22-Fountain-Blue-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/22-Fountain-Blue-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/23-Biscayne-Blue-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/23-Biscayne-Blue-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/24-Limelight-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/24-Limelight-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg'], 'solid': ['/vossen-felgen/angled/25-Gloss-Black-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/25-Gloss-Black-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/26-Satin-Black-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/26-Satin-Black-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/27-Matte-Black-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/27-Matte-Black-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/28-Textured-Black-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/28-Textured-Black-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/29-Gloss-Charcoal-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/29-Gloss-Charcoal-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/30-Gloss-Gunmetal-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/30-Gloss-Gunmetal-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/31-Matte-Gunmetal-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/31-Matte-Gunmetal-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/32-Textured-Gunmetal-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/32-Textured-Gunmetal-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/33-Gloss-Silver-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/33-Gloss-Silver-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/34-Satin-Silver-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/34-Satin-Silver-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/35-Pewter-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/35-Pewter-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/36-Ballteship-Grey-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/36-Ballteship-Grey-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/37-Espresso-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/37-Espresso-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/38-Gloss-Bronze-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/38-Gloss-Bronze-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/39-Satin-Bronze-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/39-Satin-Bronze-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/40-Textured-Bronze-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/40-Textured-Bronze-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/41-Butterscotch-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/41-Butterscotch-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/42-Heritage-Gold-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/42-Heritage-Gold-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/43-Gloss-Gold-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/43-Gloss-Gold-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/44-Satin-Gold-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/44-Satin-Gold-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/45-Denim-Blue-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/45-Denim-Blue-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/46-Ultraviolet-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/46-Ultraviolet-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/47-Fire-Red-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/47-Fire-Red-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg', '/vossen-felgen/angled/48-Miami-White-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1002-400x400.jpg', '/vossen-felgen/head-on/48-Miami-White-Vossen-Forged-Finishing-Options--Vossen-Wheels-2015-1001-400x400.jpg']}