Get all the product's href from AliExpress using selenium-CodePudding

I want to get and print all the Urls of the products in one of the AliExpress pages, it worked but just the first 12 urls that show and not all the urls on the page, normally it should be 60 urls because every page in AliExpress contains 60 products. How can I extract all the 60 urls. Any help is highly appreciated.

This is my code :

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from time import sleep

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.get("https://www.aliexpress.com/wholesale?trafficChannel=main&d=y&CatId=0&SearchText=smart lock&ltype=wholesale&SortType=total_tranpro_desc")
urls = [a.get_attribute('href') for a in driver.find_elements(By.CSS_SELECTOR,'a[href*="/item/"]')]
for url in urls:    
    print(url)

CodePudding user response：

The web page there not presenting all the products. It uses specific loading strategy to present only currently visible products.
To grab all the products you will have to scroll the page collecting the presented products during the scrolling. The URLs should be stored in set to avoid duplicates.
I tried the following simple solution and it worked for me:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
import time

options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(service=webdriver_service, options=options)
url = "https://www.aliexpress.com/wholesale?trafficChannel=main&d=y&CatId=0&SearchText=smart lock&ltype=wholesale&SortType=total_tranpro_desc"

driver.get(url)
urls = set()
for i in range(5):
    block = [a.get_attribute('href') for a in driver.find_elements(By.CSS_SELECTOR, 'a[href*="/item/"]')]
    urls.update(block)
    driver.execute_script("window.scrollBy(0, arguments[0]);", 600)
    time.sleep(2)
for url in urls:
    print(url)

The output is:

https://he.aliexpress.com/item/1005004505011076.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-33&pdp_ext_f={"sku_id":"12000029397314035"}&pdp_npi=2@dis!ILS!680.47!238.16!!!!!@0b0a0ac216625452616837499e4d53!12000029397314035!sea&curPageLogUid=eOPom12i1YdL
https://he.aliexpress.com/item/1005003118129589.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-3&pdp_ext_f={"sku_id":"12000024195825216"}&pdp_npi=2@dis!ILS!17.75!14.38!!!0.0!!@0b0a0ac216625452616837499e4d53!12000024195825216!sea&curPageLogUid=b4KiLx28AEal
https://he.aliexpress.com/item/1005004314575646.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-27&pdp_ext_f={"sku_id":"12000028723076890"}&pdp_npi=2@dis!ILS!626.53!307.0!!!!!@0b0a0ac216625452616837499e4d53!12000028723076890!sea&curPageLogUid=iOoGBaB9fk1i
https://he.aliexpress.com/item/4000154297870.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-4&pdp_ext_f={"sku_id":"12000017734780587"}&pdp_npi=2@dis!ILS!76.86!39.18!!!7.4!!@0b0a0ac216625452616837499e4d53!12000017734780587!sea&curPageLogUid=iMbrWCxxlNRK
https://he.aliexpress.com/item/1005002868522660.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-28&pdp_ext_f={"sku_id":"12000022534558777"}&pdp_npi=2@dis!ILS!63.46!38.69!!!21.15!!@0b0a0ac216625452616837499e4d53!12000022534558777!sea&curPageLogUid=kzYKVU82QxVZ
https://he.aliexpress.com/item/1005004495398206.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-44&pdp_ext_f={"sku_id":"12000029361338239"}&pdp_npi=2@dis!ILS!303.25!239.56!!!!!@0b0a0ac216625452616837499e4d53!12000029361338239!sea&curPageLogUid=ZCNQVDFjiW24
https://he.aliexpress.com/item/1005002843575207.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-30&pdp_ext_f={"sku_id":"12000026607998519"}&pdp_npi=2@dis!ILS!294.41!232.57!!!!!@0b0a0ac216625452616837499e4d53!12000026607998519!sea&curPageLogUid=dKGPq84ciu6A
https://he.aliexpress.com/item/4001166182060.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-14&pdp_ext_f={"sku_id":"10000014983376216"}&pdp_npi=2@dis!ILS!83.98!67.17!!!3.51!!@0b0a0ac216625452616837499e4d53!10000014983376216!sea&curPageLogUid=5P64jj4BsxMb
https://he.aliexpress.com/item/1005002864846136.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-31&pdp_ext_f={"sku_id":"12000027311675115"}&pdp_npi=2@dis!ILS!321.74!254.18!!!30.24!!@0b0a0ac216625452616837499e4d53!12000027311675115!sea&curPageLogUid=wIquG44FDWvS
https://he.aliexpress.com/item/1005004407267255.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-25&pdp_ext_f={"sku_id":"12000029260939933"}&pdp_npi=2@dis!ILS!1117.23!893.79!!!!!@0b0a0ac216625452616837499e4d53!12000029260939933!sea&curPageLogUid=L9QVMI7Dkc9L
https://he.aliexpress.com/item/1005003202983418.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-13&pdp_ext_f={"sku_id":"12000024642176626"}&pdp_npi=2@dis!ILS!76.75!27.75!!!!!@0b0a0ac216625452616837499e4d53!12000024642176626!sea&curPageLogUid=w934YlWjZpSk
https://he.aliexpress.com/item/10000345027263.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-40&pdp_ext_f={"sku_id":"20000000178547076"}&pdp_npi=2@dis!ILS!73.66!47.88!!!6.31!!@0b0a0ac216625452616837499e4d53!20000000178547076!sea&curPageLogUid=1CXW2eZmYyoz
https://he.aliexpress.com/item/1005003782715703.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-41&pdp_ext_f={"sku_id":"12000028520788363"}&pdp_npi=2@dis!ILS!733.13!403.22!!!!!@0b0a0ac216625452616837499e4d53!12000028520788363!sea&curPageLogUid=DzR0GUWN6DqQ
https://he.aliexpress.com/item/1005003093642821.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-11&pdp_ext_f={"sku_id":"12000025801875061"}&pdp_npi=2@dis!ILS!823.63!271.78!!!!!@0b0a0ac216625452616837499e4d53!12000025801875061!sea&curPageLogUid=yVO5hwgoHmV1
https://he.aliexpress.com/item/4000918150378.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-23&pdp_ext_f={"sku_id":"12000021385410206"}&pdp_npi=2@dis!ILS!178.69!146.52!!!86.4!!@0b0a0ac216625452616837499e4d53!12000021385410206!sea&curPageLogUid=C6dkvTMogJ62
https://he.aliexpress.com/item/1005001900549701.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-16&pdp_ext_f={"sku_id":"12000024274381461"}&pdp_npi=2@dis!ILS!188.58!105.62!!!!!@0b0a0ac216625452616837499e4d53!12000024274381461!sea&curPageLogUid=d1dAHgBFOsqF
https://he.aliexpress.com/item/1005003257440167.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-29&pdp_ext_f={"sku_id":"12000029883679176"}&pdp_npi=2@dis!ILS!891.4!410.06!!!!!@0b0a0ac216625452616837499e4d53!12000029883679176!sea&curPageLogUid=D3UOiF7Eswwj
https://he.aliexpress.com/item/33023975249.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-19&pdp_ext_f={"sku_id":"12000026754718473"}&pdp_npi=2@dis!ILS!132.1!47.57!!!12.0!!@0b0a0ac216625452616837499e4d53!12000026754718473!sea&curPageLogUid=iVmMqv3Hv2dw
https://he.aliexpress.com/item/1005004105323148.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-35&pdp_ext_f={"sku_id":"12000028159854219"}&pdp_npi=2@dis!ILS!497.87!388.34!!!!!@0b0a0ac216625452616837499e4d53!12000028159854219!sea&curPageLogUid=ScGffoZSAzHu
https://he.aliexpress.com/item/1005003076385676.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-38&pdp_ext_f={"sku_id":"12000023907005619"}&pdp_npi=2@dis!ILS!86.71!24.94!!!!!@0b0a0ac216625452616837499e4d53!12000023907005619!sea&curPageLogUid=QArTgWyf3MS7
https://he.aliexpress.com/item/1005003731084162.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-2&pdp_ext_f={"sku_id":"12000029315337088"}&pdp_npi=2@dis!ILS!743.72!327.24!!!!!@0b0a0ac216625452616837499e4d53!12000029315337088!sea&curPageLogUid=aMsEw0Z36uSN
https://he.aliexpress.com/item/4001010240435.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-43&pdp_ext_f={"sku_id":"10000013511866229"}&pdp_npi=2@dis!ILS!29.36!3.02!!!!!@0b0a0ac216625452616837499e4d53!10000013511866229!sea&curPageLogUid=z1vD7tD9W5MZ
https://he.aliexpress.com/item/4000154478484.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-8&pdp_ext_f={"sku_id":"12000021350825852"}&pdp_npi=2@dis!ILS!78.64!35.39!!!7.4!!@0b0a0ac216625452616837499e4d53!12000021350825852!sea&curPageLogUid=yRWy6gHc4b1E
https://he.aliexpress.com/item/1005003346086758.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-32&pdp_ext_f={"sku_id":"12000025331814334"}&pdp_npi=2@dis!ILS!95.31!58.12!!!!!@0b0a0ac216625452616837499e4d53!12000025331814334!sea&curPageLogUid=iGvJbXPDD3cg
https://he.aliexpress.com/item/4000065467682.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-6&pdp_ext_f={"sku_id":"10000015061976915"}&pdp_npi=2@dis!ILS!281.64!191.53!!!162.48!!@0b0a0ac216625452616837499e4d53!10000015061976915!sea&curPageLogUid=1qiD9mBYf6KM
https://he.aliexpress.com/item/1005002676337744.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-1&pdp_ext_f={"sku_id":"12000021669614390"}&pdp_npi=2@dis!ILS!561.28!392.91!!!!!@0b0a0ac216625452616837499e4d53!12000021669614390!sea&curPageLogUid=Y1Mvi9ey1Hwl
https://he.aliexpress.com/item/4000834334480.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-34&pdp_ext_f={"sku_id":"12000028610539908"}&pdp_npi=2@dis!ILS!162.83!138.42!!!68.96!!@0b0a0ac216625452616837499e4d53!12000028610539908!sea&curPageLogUid=6CijCm642BQh
https://he.aliexpress.com/item/4001031327478.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-7&pdp_ext_f={"sku_id":"12000015965032899"}&pdp_npi=2@dis!ILS!1037.26!383.79!!!!!@0b0a0ac216625452616837499e4d53!12000015965032899!sea&curPageLogUid=fLjjoL5HUrxi
https://he.aliexpress.com/item/1005004262179786.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-12&pdp_ext_f={"sku_id":"12000028573420101"}&pdp_npi=2@dis!ILS!679.22!319.24!!!!!@0b0a0ac216625452616837499e4d53!12000028573420101!sea&curPageLogUid=n4io2MIKAEsP
https://he.aliexpress.com/item/1005003515525914.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-0&pdp_ext_f={"sku_id":"12000026134836061"}&pdp_npi=2@dis!ILS!493.37!360.15!!!!!@0b0a0ac216625452616837499e4d53!12000026134836061!sea&curPageLogUid=qY8tyv2zwcdK
https://he.aliexpress.com/item/1005003230965691.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-26&pdp_ext_f={"sku_id":"12000024764885745"}&pdp_npi=2@dis!ILS!188.3!133.68!!!!!@0b0a0ac216625452616837499e4d53!12000024764885745!sea&curPageLogUid=CJHjTgOCDRJ3
https://he.aliexpress.com/item/4000975407287.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-10&pdp_ext_f={"sku_id":"10000015021478939"}&pdp_npi=2@dis!ILS!680.62!299.47!!!!!@0b0a0ac216625452616837499e4d53!10000015021478939!sea&curPageLogUid=WDdJT8cOU94O
https://he.aliexpress.com/item/1005003187314155.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-42&pdp_ext_f={"sku_id":"12000024567051659"}&pdp_npi=2@dis!ILS!157.54!94.54!!!!!@0b0a0ac216625452616837499e4d53!12000024567051659!sea&curPageLogUid=4qQvpAeygSFw
https://he.aliexpress.com/item/32954893736.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-39&pdp_ext_f={"sku_id":"10000002371991125"}&pdp_npi=2@dis!ILS!303.58!100.18!!!!!@0b0a0ac216625452616837499e4d53!10000002371991125!sea&curPageLogUid=yDxMTsGMA1m1
https://he.aliexpress.com/item/1005002548126691.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-15&pdp_ext_f={"sku_id":"12000021089170892"}&pdp_npi=2@dis!ILS!71.49!18.2!!!!!@0b0a0ac216625452616837499e4d53!12000021089170892!sea&curPageLogUid=xfzmTqNWeUEV
https://he.aliexpress.com/item/4000735115535.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-17&pdp_ext_f={"sku_id":"12000027051383811"}&pdp_npi=2@dis!ILS!223.41!176.51!!!28.2!!@0b0a0ac216625452616837499e4d53!12000027051383811!sea&curPageLogUid=FasdXFlpMCuE
https://he.aliexpress.com/item/1005002659060326.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-20&pdp_ext_f={"sku_id":"12000021594037009"}&pdp_npi=2@dis!ILS!1144.03!800.83!!!!!@0b0a0ac216625452616837499e4d53!12000021594037009!sea&curPageLogUid=qWxA5NkkHX7u
https://he.aliexpress.com/item/1005003050882550.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-21&pdp_ext_f={"sku_id":"12000023509149328"}&pdp_npi=2@dis!ILS!78.01!35.5!!!!!@0b0a0ac216625452616837499e4d53!12000023509149328!sea&curPageLogUid=eazJyQ2xYlTa
https://he.aliexpress.com/item/1005001429138105.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-5&pdp_ext_f={"sku_id":"12000016106374836"}&pdp_npi=2@dis!ILS!908.29!272.49!!!!!@0b0a0ac216625452616837499e4d53!12000016106374836!sea&curPageLogUid=UTqfAcwKjxt3
https://he.aliexpress.com/item/1005003575980819.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-9&pdp_ext_f={"sku_id":"12000026346773711"}&pdp_npi=2@dis!ILS!56.48!37.29!!!!!@0b0a0ac216625452616837499e4d53!12000026346773711!sea&curPageLogUid=t6N9K9ie8MII
https://he.aliexpress.com/item/1005004406234563.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-24&pdp_ext_f={"sku_id":"12000029068345270"}&pdp_npi=2@dis!ILS!334.62!167.31!!!19.93!!@0b0a0ac216625452616837499e4d53!12000029068345270!sea&curPageLogUid=ERz9XojcigBs
https://he.aliexpress.com/item/1005003208283080.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-18&pdp_ext_f={"sku_id":"12000024667248681"}&pdp_npi=2@dis!ILS!164.38!100.25!!!!!@0b0a0ac216625452616837499e4d53!12000024667248681!sea&curPageLogUid=Z5qyCqcRmY48
https://he.aliexpress.com/item/1005002983524272.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-37&pdp_ext_f={"sku_id":"12000023060725911"}&pdp_npi=2@dis!ILS!164.45!146.35!!!!!@0b0a0ac216625452616837499e4d53!12000023060725911!sea&curPageLogUid=LPz7kSEHNr3P
https://he.aliexpress.com/item/32964505971.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-22&pdp_ext_f={"sku_id":"12000020850078768"}&pdp_npi=2@dis!ILS!861.0!266.91!!!!!@0b0a0ac216625452616837499e4d53!12000020850078768!sea&curPageLogUid=ehbKLbbMGsal
https://he.aliexpress.com/item/1005004243027957.html?algo_pvid=cf9a5cc9-df53-404b-a8ed-db8a227069e6&algo_exp_id=cf9a5cc9-df53-404b-a8ed-db8a227069e6-36&pdp_ext_f={"sku_id":"12000029030797848"}&pdp_npi=2@dis!ILS!723.76!325.69!!!!!@0b0a0ac216625452616837499e4d53!12000029030797848!sea&curPageLogUid=SAk2CRsHhKDu

Process finished with exit code 0