Selenium not able to locate the element-CodePudding

I'm trying to scrape data from https://in.puma.com/in/en/mens/mens-new-arrivals . The complete data is loaded when the show all button is clicked.

I used selenium to generate the click and load the rest of the page, however - I'm getting an error

"TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: "

See my code below.

from selenium.webdriver.support import expected_conditions as EC
import time
from lxml import etree as et

chrome_driver_path = "driver/chromedriver"

url = 'https://in.puma.com/in/en/mens/mens-new-arrivals'
browser = webdriver.Chrome(ChromeDriverManager().install())
browser.get(url)

x_path_to_load_more = '//*[@data-component-id="a_tspn9cqoeth"]'
browser.execute_script("window.scrollTo(0,document.body.scrollHeight)")
button_locate = wait(browser,10).until(EC.presence_of_element_located((By.XPATH,x_path_to_load_more)))
button_locate.click()

CodePudding user response：

The xpath is not correct, try

x_path_to_load_more = "//button[contains(., 'Show All')]"

To verify the effectiveness of the xpath inspect the page, open the find bar with Command F or Control F and paste your xpath

CodePudding user response：

@data-component-id seem to be dynamic. It means that the value will be different (not "a_tspn9cqoeth") each time you open that page. Try to search by another attribiute value:

x_path_to_load_more = '//div[@]/button[contains(@class, "show-more-button")]'

x_path_to_load_all = '//div[@]/button[contains(@class, "show-all-button")]'

Also it's better to use EC.element_to_be_clickable instead of EC.presence_of_element_located

UPDATE

Since click on button might be intercepted by Cookies footer try to scroll page down before making click:

from selenium.webdriver.common.keys import Keys
driver.find_element('xpath', '//body').send_keys(Keys.END)