I have to collect data from this link: https://datacentersupport.lenovo.com/gb/en/products/storage/fibre-channel-switches/b6505-fc-san-switch/3873/parts/display/compatible

However I am having difficulty accessing the data under substitutes(note: not all them have substitutes). Those with substitutes look like this:

An example is on page 2 of the link

Kindly help me finish the code I have to collect the part numbers of the substitutes.

Here's my code:

from selenium import webdriver
from time import sleep
import csv

# initializing webdriver 
driver = webdriver.Chrome(executable_path="~~chromedriver.exe")
url = "https://datacentersupport.lenovo.com/gb/en/products/storage/fibre-channel-switches/b6505-fc-san-switch/3873/parts/display/compatible"
driver.get(url)
sleep(5)

#getting breadcrumbs
bread1 = driver.find_element_by_xpath("//span[@class='prod-catagory-name']")
bread2 = driver.find_element_by_xpath("//span[@class='prod-catagory-name']/a")

#grabbing table data and navigating 
pages = int(driver.find_element_by_xpath("//div[@class='page-container']/span[@class='icon-s-right active']/preceding-sibling::span[1]").text)
num = pages -1 
for _ in range(num):
     rows = driver.find_elements_by_xpath("//table/tbody/tr/td[2]/div")
     for row in rows:
         parts = row.text
         with open(filename, 'a', encoding='utf-8') as file:
             file.write(url   ","   bread1.text   ","   bread2.text   ","   parts   "\n")

     pagination = driver.find_element_by_xpath("//div[@class='pagecontainer']/span[@class='icon-s-right active']").click()
     sleep(5)
driver.close()

Please let me know if I need to change or modify the code to get the substitutes.

CodePudding user response：

To access the data under substitutes within the webpage you need to expand the Substitutes inducing WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

• Code Block:

  driver.get("https://datacentersupport.lenovo.com/gb/en/products/storage/fibre-channel-switches/b6505-fc-san-switch/3873/parts/display/compatible")
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "span.icon-select-down"))).click()
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[contains(., '20')]"))).click()
  for element in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "span[class='icon-s-down']"))):
      element.click()
      print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[@class='icon-s-up']//following::tr[3]/td[contains(@class,'enabled-border')]//div[text()]"))).text)
  

• Console Output:

  Lenovo 30m LC-LC OM3 MMF Cable
  Lenovo 30m LC-LC OM3 MMF Cable
  5m LC-LC OM3 MMF Cable
  5m LC-LC OM3 MMF Cable