I am trying to click load more button infinite time until load all products. But my load more button not working. here is my code:

url = 'https://www.pursemall.ru/vuitton/damier-azur'
driver = webdriver.Chrome()
driver.maximize_window()
driver.get(url)


try:
    for i in range(1000):
     load_more_button = driver.find_element_by_xpath("//span[text()='Load More Products']")
     load_more_button.click()
except:
    pass
    print("task load more button completed") 

why my load more button not working? where I am doing mistake?

CodePudding user response：

What you can do is the below. Each call inside the loop will fetch a set of products. You can scrape each response and collect the data.

In the browser do F12 -> Network -> XHR and see how the browser does the HTTP call every time you hit Load More Products

import requests


for page in range(1,4):
    url = f'https://www.pursemall.ru/vuitton/damier-azur?page={page}'
    print(url)
    r = requests.get(url)
    print(r.status_code)

CodePudding user response：

I'm not 100% sure why your original code wasn't functioning, but I was able to get it working by copying the full XPATH of the button instead of what you have in your code above.

I also used WebDriverWait and sleep() to wait a few seconds until the button was loaded after each click, since that's good practice for web scrapers. Perhaps the issue was a combination of the two :)

url = 'https://www.pursemall.ru/vuitton/damier-azur'
driver = webdriver.Chrome(PATH) # PATH is where you have chromedriver stored on your machine
driver.maximize_window()
driver.get(url)

try:
    for i in range(1000):
        load_more_button = WebDriverWait(driver, 10).until(ec.element_to_be_clickable(
            (By.XPATH, '/html/body/div[1]/div/div[3]/div[3]/span/a/span')))
        sleep(3)
        load_more_button.click()
except:
    pass
    print("task load more button completed")