Why is my code not printing the product links?-CodePudding

I have written a script to try and print all of the individual links for items listed on a particular web page. Here is the code so far:

from selenium.webdriver.common.by import By
from selenium import webdriver
from selenium.webdriver.chrome.options import Options


options=Options()

driver=webdriver.Chrome(options=options)

driver.get('https://www.lazada.com.ph/shop-portable-speakers/?spm=a2o4l.home.cate_2_2.2.239e359dxynAFV')

link_elements=driver.find_elements(By.NAME,'product-item')

links = []
for link_el in link_elements:
    href = link_el.get_attribute("href")
    print (href)
    links.append(href)
driver.quit()

Any ideas as to what I am doing wrong here?.

CodePudding user response：

You instantly try and search for 'product-item' before the webpage has a chance to fully load. Wait until the webpage is loaded. Add a time.sleep() after your driver.get()

import time

sec = 5

time.sleep(sec)

Or you can wait for a specific element to load on the webpage using the following:

# New Imports to add
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

# Create a wait object with a specified maximum time to wait
maxTimeToWait = 20
wait = WebDriverWait(driver, maxTimeToWait)

driver.get('https://www.lazada.com.ph/shop-portable-speakers/?spm=a2o4l.home.cate_2_2.2.239e359dxynAFV')

# This will wait for the specified element to appear, but note just because this element has loaded does not mean that the elements below this element have loaded
link_element = wait.until(EC.presence_of_element_located((By.NAME, 'product-item')

# Then find all of the elements with this name
link_elements=driver.find_elements(By.NAME,'product-item')

CodePudding user response：

The product links are contained within the src attribute. As an example:

<img age="0" type="product"  alt="LEERFEI E-91 bluetooth Speaker Sound bar with subwoofer Sound card With USB Por TF AUX Smart Sound bar TV Speaker Home Theatre For TV PC" src="https://lzd-img-global.slatic.net/g/p/b5e64a3099b0b8d7ca36986909c1e213.jpg_200x200q80.jpg_.webp" style="object-fit: fill;">

Solution

To print the value of the src attributes you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

driver.get("https://www.lazada.com.ph/shop-portable-speakers/?spm=a2o4l.home.cate_2_2.2.239e359dxynAFV")
print([my_elem.get_attribute("src") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.picture-wrapper>img[type='product']")))])

Using XPATH in a single line:

driver.get("https://www.lazada.com.ph/shop-portable-speakers/?spm=a2o4l.home.cate_2_2.2.239e359dxynAFV")
print([my_elem.get_attribute("src") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='picture-wrapper']/img[@type='product']")))])

Console Output:

['https://lzd-img-global.slatic.net/g/p/4cdcf0125c301266fe7c0e1004b5a435.png_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/0c8b59946bbf796b0cf3931a96c6370a.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/5b5fc2ed633bd48c1fb2e2c4312756f8.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/486c93c9d03ad47268307f1073286baf.png_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/b06bd4c0b78b4f4d0b8026392f8f39cd.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/15224b1fb1446eddf2a95afb23cf7046.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/cff844c6e38761d956478104970a3e16.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/ca0eb3ba62e97874f06752b6b3611e34.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/b45fd9513f86efcf66a947286f314133.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/0a9dcda2aaac65996b5fa7bc35d40d7d.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/648aeb1dff5c0ea8e33ef86baa20881e.png_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/8625845c2d11c4b9613b1db899d109f4.png_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/b5627d2427021e81596c2fe0dcfc485c.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/49b4c1168e6145056a5e5f7a2a7aa9dd.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/8b58661402a26f48373e8b767f4768e9.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/d273ecbdebdecd893a057ab532baa248.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/6618804d6ab326773e402def53f0b378.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/d38d70e2650b74ce1f07767878263027.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/c88b9713573065ce7779e1a4020248b0.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/f16c7a8d201eb17b3d6b2692f8d22974.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/0bb3e0090c1fd561ec5a4c19316c2afb.png_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/f11b1eb2c12d4919fa39c39491b4992d.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/b76540bea6d569b181250f98bb5ea0e8.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/d1694d4083e46ad2c227dc848980269c.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/cc91211da528c101dbd891baaf1ff13c.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/231adb730f9f0d4629ac3801d7eed7a6.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/5e6b63361ff76713b594e03d433b20c6.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/84e864e4ed14b2ee69538dac74f35355.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/0611a9678108c1de7765fefd3ccc0232.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/ca47f523b0dfdc52afb6b99dae25c8a0.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/d7c900d8e3c1d0d3f1c112949c352d30.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/fe77cf1c0cb75d2c5ac56304ab694b91.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/b18241a2b7ff47ad06109ee9e87bc5c7.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/b64da52a80bde951e7b0667b09eef892.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/3dcbf8068675d4a6f22fc38a8efe3b94.png_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/2072df22427ed5d5cfed506c1d420cb3.png_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/833fa8fa0121d983e131fdf12b6528aa.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/9e2f8db84dc245bb3e7f582953a19499.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/ec98df1bdaf343709b76fab7ea8aa083.jpg_200x200q80.jpg_.webp', 'https://lzd-img-global.slatic.net/g/p/7b36600fccf1256b9402e886364820a9.jpg_200x200q80.jpg_.webp']

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC