selenium by_xpath not returning any results-CodePudding

I am using Selenium 4 , and I seem to not get back the any result when requesting for elements in a div.


# Wait for the page to load
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.ID, "search-key")))

# wait for the page to load
driver.implicitly_wait(10)

# search for the suppliers you want to message
search_input = driver.find_element(By.ID,"search-key")
search_input.send_keys("suppliers")
search_input.send_keys(Keys.RETURN)

# find all the supplier stores on the page
supplier_stores_div = driver.find_element(By.CLASS_NAME, "list--gallery--34TropR")



print(supplier_stores_div)

supplier_stores = supplier_stores_div.find_elements(By.XPATH, "./a[@class='v3--container--31q8BOL cards--gallery--2o6yJVt']")

print(supplier_stores)

The logging statements gave me <selenium.webdriver.remote.webelement.WebElement (session="a3be4d8c5620e760177247d5b8158823", element="5ae78693-4bae-4826-baa6-bd940fa9a41b")> and an empty list for the Element objects, []

The html code is here:

<div  data-spm="main" data-spm-max-idx="24">flex
                                                                               
  <a  href="(link)" target="_blank" style="text-align: left;" data-spm-anchor-id="a2g0o.productlist.main.1"> (some divs) </a>flex

That is just one <a> class, there's more.

CodePudding user response：

It returns you the Element object. If you want to get the text you need to write

supplier_stores_div.find_elements(By.XPATH, "./a[@class='v3--container--31q8BOL cards--gallery--2o6yJVt']").getText()

or for getting an attribute (for example the href)

supplier_stores_div.find_elements(By.XPATH, "./a[@class='v3--container--31q8BOL cards--gallery--2o6yJVt']").getAttribute("href")

CodePudding user response：

Before scraping the supplier names, you have to scroll down the page slowly to the bottom, then only you can get all the supplier names, try the below code:

driver.get("https://www.aliexpress.com/premium/supplier.html?spm=a2g0o.best.1000002.0&initiative_id=SB_20221218233848&dida=y")

last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    driver.execute_script("window.scrollBy(0, 800);")
    sleep(1)
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

suppliers = driver.find_elements(By.XPATH, ".//*[@class='list--gallery--34TropR']//span/a")
print("Total no. of suppliers:", len(suppliers))
print("======")
for supplier in suppliers:
    print(supplier.text)

Output:

Total no. of suppliers: 60
======
Reading Life Store
ASONSTEEL Store
Paper, ink, pen and inkstone Store
Custom Stationery Store
ZHOUYANG Official Store
WOWSOCOOL Store
IFPD Official Store
The 9 Store
QuanRun Store
...
...