I am trying to click on the links for each of the product tiles
My current code: `
import numpy as np
import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
import time
driverfile = r'C:\Users\Main\Documents\Work\Projects\Scraping Websites\extra\chromedriver'
driver = webdriver.Chrome(executable_path=driverfile)
driver.implicitly_wait(10)
url = "https://www.hugoboss.com/uk/men-clothing/"
driver.get(url)
driver.implicitly_wait(10)
shadowRoot = driver.find_element(By.XPATH,"//div[@id='usercentrics-root']").shadow_root
shadowRoot.find_element(By.CSS_SELECTOR, "button[data-testid='uc-save-button']").click()
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-testid='uc-save-button']"))).click()
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='product-tile-plp__title-link.font--sub2.js-product-tile-link.widget-initialized']"))).click()
` Current Error:
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-testid='uc-save-button']"))).click()
File "C:\Users\Main\Anaconda3\lib\site-packages\selenium\webdriver\support\wait.py", line 87, in until
raise TimeoutException(message, screen, stacktrace)
TimeoutException
CodePudding user response:
Actually, you can't click on each link from title because they didn't contain any clickable button but you can iterate them to pull the url. I use bs4 to grab each link because they aren't dynamic.
from bs4 import BeautifulSoup
import requests
URL = 'https://www.hugoboss.com/uk/men-clothing/'
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'lxml')
for link in soup.select('.product-tile-plp__title a'):
link = 'https://www.hugoboss.com' link.get('href')
print(link)
Output:
https://www.hugoboss.com/uk/three-pack-of-regular-fit-cotton-t-shirts/hbeu50325887_961.html
https://www.hugoboss.com/uk/slim-fit-shorts-in-stretch-cotton-twill/hbeu50467083_037.html
https://www.hugoboss.com/uk/slim-fit-three-piece-suit-in-stretch-wool/hbeu50478270_424.html
https://www.hugoboss.com/uk/organic-cotton-polo-shirt-with-curved-logo/hbeu50468983_453.html
https://www.hugoboss.com/uk/slim-fit-trousers-in-stretch-cotton-satin/hbeu50470813_404.html
https://www.hugoboss.com/uk/slim-fit-suit-in-micro-patterned-traceable-stretch-wool/hbeu50468911_413.html
https://www.hugoboss.com/uk/slim-fit-suit-in-micro-patterned-performance-stretch-fabric/hbeu50474242_273.html
https://www.hugoboss.com/uk/organic-cotton-polo-shirt-with-curved-logo/hbeu50468983_401.html
https://www.hugoboss.com/uk/slim-fit-shorts-in-structured-stretch-cotton/hbeu50472870_404.htmlhttps://www.hugoboss.com/uk/tapered-fit-chinos-in-overdyed-stretch-cotton-satin/hbeu50470797_404.html
https://www.hugoboss.com/uk/three-pack-of-regular-fit-cotton-t-shirts/hbeu50325887_975.html
https://www.hugoboss.com/uk/three-pack-of-regular-fit-cotton-t-shirts/hbeu50325887_974.html
https://www.hugoboss.com/uk/slim-fit-shorts-in-stretch-cotton-twill/hbeu50467083_404.html
https://www.hugoboss.com/uk/slim-fit-suit-in-stretch-wool-with-logo-lining/hbeu50474754_068.html
https://www.hugoboss.com/uk/single-breasted-jacket-in-virgin-wool-serge/hbeu50469172_401.html
https://www.hugoboss.com/uk/formal-trousers-in-virgin-wool-serge/hbeu50469174_401.html
https://www.hugoboss.com/uk/slim-fit-shorts-in-structured-stretch-cotton/hbeu50472870_275.htmlhttps://www.hugoboss.com/uk/slim-fit-shirt-in-easy-iron-cotton-poplin/hbeu50289499_199.html
https://www.hugoboss.com/uk/slim-fit-trousers-in-stretch-cotton-satin/hbeu50470813_239.html
https://www.hugoboss.com/uk/slim-fit-trousers-in-stretch-cotton-satin/hbeu50470813_027.html
https://www.hugoboss.com/uk/single-breasted-jacket-in-virgin-wool-serge/hbeu50469171_401.html
https://www.hugoboss.com/uk/tuxedo-jacket-in-virgin-wool-serge/hbeu50469191_401.html
https://www.hugoboss.com/uk/tuxedo-trousers-in-virgin-wool-serge/hbeu50469186_001.html
https://www.hugoboss.com/uk/regular-fit-jeans-in-dark-blue-comfort-stretch-denim/hbeu50470509_420.html
https://www.hugoboss.com/uk/stretch-cotton-t-shirt-with-contrast-logo/hbeu50469057_402.html
https://www.hugoboss.com/uk/regular-fit-jersey-shirt-with-button-down-collar/hbeu50469854_411.html
https://www.hugoboss.com/uk/cotton-jersey-regular-fit-t-shirt-with-collaborative-artwork/hbeu50472110_001.html
https://www.hugoboss.com/uk/regular-fit-jersey-shirt-with-button-down-collar/hbeu50469854_100.html
https://www.hugoboss.com/uk/regular-fit-jersey-shirt-with-button-down-collar/hbeu50469854_453.html
https://www.hugoboss.com/uk/slim-fit-trousers-in-stretch-cotton-satin/hbeu50470813_012.html
https://www.hugoboss.com/uk/cotton-blend-slim-fit-polo-shirt-with-contrast-trims/hbeu50466442_402.html
https://www.hugoboss.com/uk/slim-fit-shorts-in-structured-stretch-cotton/hbeu50472870_337.htmlhttps://www.hugoboss.com/uk/tapered-fit-jeans-in-dark-blue-super-stretch-denim/hbeu50471005_417.html
https://www.hugoboss.com/uk/organic-cotton-polo-shirt-with-curved-logo/hbeu50468983_316.html
https://www.hugoboss.com/uk/regular-fit-suit-in-super-flex-wool-blend-cloth/hbeu50466013_001.html
https://www.hugoboss.com/uk/tuxedo-jacket-in-responsible-virgin-wool/hbeu50469185_001.html