Extract information from products on Website after pressing "more items" button with Selen-CodePudding

I managed to extract the names, specs, prices, and priceUnits from the products on this page: https://www.bauhaus.info/baustoffe/c/10000819.

I do, however, only manage to get the first 36 products visible on the page. How would I extract all the products on this page that appear when pressing on the button for "more items"?

For this, see the inspection of the page here:

see inspect here

Any help is very much appreciated!

This is my code:

from selenium import webdriver
import pandas as pd
import re

browser = webdriver.Chrome(r'C:\Users\KristerJens\Downloads\chromedriver_win32\chromedriver')


browser.get('https://www.bauhaus.info/baustoffe/c/10000819')

names= []
specs = []
prices = []
priceUnit = []

for li in browser.find_elements_by_xpath("//ul[@class='product-list-tiles row list-unstyled']/li"):
    names.append(li.find_element_by_class_name("product-list-tile__info__name").text)
    specs.append(li.find_element_by_class_name("product-list-tile__info__attributes").text)
    prices.append(li.find_element_by_class_name("price-tag__box").text.split('\n')[0]   "€")
    
    p = li.find_element_by_class_name("price-tag__sales-unit").text.split('\n')[0]
    priceUnit.append(p[p.find("(") 1:p.find(")")])
    
df2 = pd.DataFrame()
df2['names'] = names
df2['specs'] = specs
df2['prices'] = prices
df2['priceUnit'] = priceUnit

CodePudding user response：

firstly try to click on "More Products" button until it gets disabled i.e all products gets listed down and then use the common xpath for locating product info.

CodePudding user response：

For each page add scroll to element more items and click it, see below example of scroll to element implementation

from selenium.webdriver.common.action_chains import ActionChains

element = driver.find_element_by_id("more_items") 

actions = ActionChains(driver)
actions.move_to_element(element).perform()

CodePudding user response：

Was able to click on More option continuously with below code. Try to incorporate this with your code.

# Imports Required for Explicit Waits:
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver.get("https://www.bauhaus.info/baustoffe/c/10000819")
wait = WebDriverWait(driver,30)

options = driver.find_elements_by_xpath("//ul[@class='product-list-tiles row list-unstyled']/li")
print(len(options))

#Using `Count` variable to keep track of number of times of clicking on More option. Remove the `Count` part of the code to continuously click on More option.
count = 0
try:
    while True:
        if count > 5: # Click on "More" option only for 5 times
            break
        moreoption = wait.until(EC.element_to_be_clickable((By.XPATH,"//button[@data-message='adb-show-more-products-button']")))
        driver.execute_script("arguments[0].scrollIntoView(true);",moreoption)
        driver.execute_script("window.scrollBy(0,-300);")
        time.sleep(2)
        moreoption.click()
        count  = 1
        time.sleep(2)
        options = driver.find_elements_by_xpath("//ul[@class='product-list-tiles row list-unstyled']/li")
        print(len(options))
except:
    pass