I couldn't find all 'href' by "find_elements_by_xpath" Is there another way to find data? thx
!pip install selenium
from selenium import webdriver
import time
import pandas as pd
browser = webdriver.Chrome(executable_path='./chromedriver.exe')
browser.implicitly_wait(5)
browser.get("https://tw.mall.yahoo.com/store/屈臣氏Watsons:watsons")
linkPath = '//ul[@class = "sc-eWvPqa cePswM"]/li/a'
product_links = browser.find_elements_by_xpath(linkPath)
print(product_links)
CodePudding user response：
href is attribute of the anchor tag.
so this xpath
//a
should locate all of them.
or in Selenium you can use tag_name as well. I will use XPath,
browser = webdriver.Chrome(executable_path='./chromedriver.exe') browser.get("https://tw.mall.yahoo.com/store/屈臣氏Watsons:watsons")
browser.maximize_window()
linkPath = "//a"
product_links = browser.find_elements(By.XPATH, linkPath)
print(len(product_links))
for link in product_links:
print(link.get_attribute('href'))
Output:
288
https://tw.yahoo.com/
https://tw.buy.yahoo.com/
https://tw.bid.yahoo.com/
https://tw.usedcar.yahoo.com/
https://tw.mall.yahoo.com/activity?p=mall-1-0-180921-channelgroupbuy
http://mail.yahoo.com.tw/
https://yahoomode.tumblr.com/yahooapp/
https://tw.mall.yahoo.com/
https://tw.mall.yahoo.com/search/store?p=
https://tw.user.mall.yahoo.com/my/home
https://tw.sc.mall.yahoo.com/mcart/preview
https://tw.user.mall.yahoo.com/my/order/orderList
https://login.yahoo.com/config/login?.intl=tw&.src=mktg1&done=https://tw.mall.yahoo.com/store/watsons
https://tw.user.mall.yahoo.com/my/point
https://tw.mall.yahoo.com/
https://tw.user.mall.yahoo.com/my/home
https://tw.user.mall.yahoo.com/my/order/orderList
https://tw.user.mall.yahoo.com/sc/view/home
https://tw.user.mall.yahoo.com/my/notification
https://tw.user.mall.yahoo.com/my/point
https://tw.user.mall.yahoo.com/my/order/ratingList
https://tw.user.mall.yahoo.com/my/followupStore
https://tw.user.mall.yahoo.com/my/watchlist
https://tw.user.mall.yahoo.com/my/ecoupon
https://tw.user.mall.yahoo.com/my/voucher/unused
https://tw.user.mall.yahoo.com/my/member
https://tw.user.mall.yahoo.com/my/setting
https://tw.user.mall.yahoo.com/my/customerqa
https://tw.help.yahoo.com/kb/shopping-mall-web/SLN35152.html
https://tw.mall.yahoo.com/
https://tw.mall.yahoo.com/store/watsons
https://tw.mall.yahoo.com/store/watsons
https://tw.mall.yahoo.com/store/watsons
https://tw.mall.yahoo.com/store/watsons/rating/list
https://tw.mall.yahoo.com/chat/watsons
https://tw.mall.yahoo.com/store/watsons/stIntroMgt
https://tw.mall.yahoo.com/store_vip/watsons
https://tw.mall.yahoo.com/store/watsons/stNoteMgt
https://tw.mall.yahoo.com/store/watsons/edm
https://tw.mall.yahoo.com/store/watsons
None
None
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=2689&path=2689
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=2797&path=2797
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=712&path=712
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=664&path=664
None
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=1553&path=1553
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=2505&path=2505
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=1452&path=1452
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=1060&path=1060
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=827&path=827
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=1003&path=1003
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=1024&path=1024
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=875&path=875
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=958&path=958
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=979&path=979
https://tw.mall.yahoo.com/store_vip/watsons
https://tw.mall.yahoo.com/store/watsons/promo
https://tw.rcv.mall.yahoo.com/rcv/askEcoupon?s=5Ir7dQTxCtYebEIVRr7qdbQJrQ--
https://tw.mall.yahoo.com/store/watsons/promoCode?id=407205
https://tw.mall.yahoo.com/store/watsons/amountPromo?promotion_id=3101274
https://tw.mall.yahoo.com/store/watsons/amountPromo?promotion_id=3101277
https://tw.mall.yahoo.com/store/watsons/amountPromo?promotion_id=3101328
https://tw.mall.yahoo.com/store/watsons/amountPromo?promotion_id=3101279
https://tw.mall.yahoo.com/store/watsons/amountPromo?promotion_id=3101281
https://tw.mall.yahoo.com/store/watsons/amountPromo?promotion_id=3101286
https://tw.mall.yahoo.com/store/watsons/amountPromo?promotion_id=3101276
https://tw.mall.yahoo.com/store/watsons/amountPromo?promotion_id=3101216
https://tw.mall.yahoo.com/store/watsons/amountPromo?promotion_id=3101318
https://tw.mall.yahoo.com/store/watsons/amountPromo?promotion_id=3101272
https://tw.mall.yahoo.com/store/watsons/amountPromo?promotion_id=3101228
https://tw.mall.yahoo.com/store/watsons/amountPromo?promotion_id=3101287
https://tw.mall.yahoo.com/store/watsons
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=4792&path=4792
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=4802&path=4793,4802
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=4805&path=4794,4805
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=4807&path=4794,4807
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=4799&path=4793,4799
https://tw.mall.yahoo.com/search?m=list&sid=watsons&ccatid=4801&path=4793,4801
https://tw.mall.yahoo.com/search?q=活沛多&sid=watsons
https://tw.mall.yahoo.com/store/