I'v been trying to scraping https://www.uniqlo.com/tw/zh_TW/women_tops_long-and-3-4sleeves-and-cardigan.html by selenium.
driver.get('https://www.uniqlo.com/tw/zh_TW/c/ALL020000024.html')
WebDriverWait(driver, 30, 0.5).until(
EC.presence_of_element_located((By.CLASS_NAME, 'picture-img')))
time.sleep(20)
a = driver.find_elements(By.CLASS_NAME, 'product-li')
print("a: ",a,"\n")
results = []
It has 20 products that we want. And we want to find the detail information of these products. So we use get_attribute.
for x in a:
time.sleep(10)
src = x.find_element(By.CLASS_NAME, 'picture-img').get_attribute('src')
alt = x.find_element(By.CLASS_NAME, 'picture-img').get_attribute('alt')
print("src: ",src )
print("alt: ",alt)
result = {
'src': src,
'slt': alt,
}
results.append(result)
But the loop break down. It only get the 1 to 8 products' attribute. There are actually 20 elements in list a, why we only got some of the attributes not all of them? enter image description here
CodePudding user response：
Just get the data straight from the source:
import requests
from datetime import datetime
import pandas as pd
url = 'https://www.uniqlo.com/tw/zh_TW/data/pages/women_tops_long-and-3-4sleeves-and-cardigan.html.json'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36'
}
payload = {
't':str(int(datetime.now().timestamp()))}
jsonData = requests.get(url, headers=headers, params=payload).json()
items = jsonData['section08']['props']['items']
basic_info_df = pd.DataFrame(items)
rows = []
for productCode in list(basic_info_df['productCode']):
print(productCode)
url = f'https://www.uniqlo.com/tw/data/products/prodInfo/zh_TW/{productCode}.json'
jsonData = requests.get(url, headers=headers).json()
rows.append(jsonData)
full_df = pd.DataFrame(rows)
Output:
print(full_df.to_string())
originPrice score inactive mainPic minSize stock hasStock gDeptValue priceColor colorList evaluationCount minPrice name maxPrice code maxSize enabledFlag identity approval sex fullName label timeLimitedBegin productCode timeLimitedEnd
0 990.0 0.0 N /hmall/test/u0000000011533/main/first/1000/1.jpg S Y Y 女裝 black [{'styleText': '01 OFF WHITE', 'colorNo': 'COL01'}, {'styleText': '06 GRAY', 'colorNo': 'COL06'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}] 0 990.0 特級彈性DRY T恤 S 990.0 447126 XL Y [new_product] LIST 女裝 女裝 特級彈性DRY T恤 S 447126 new_product u0000000011533
1 590.0 0.0 N /hmall/test/u0000000011620/main/first/1000/1.jpg XS Y Y 女裝 black [{'styleText': '01 OFF WHITE', 'colorNo': 'COL01'}, {'styleText': '08 DARK GRAY', 'colorNo': 'COL08'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}, {'styleText': '10 PINK', 'colorNo': 'COL10'}, {'styleText': '30 NATURAL', 'colorNo': 'COL30'}, {'styleText': '50 LIGHT GREE', 'colorNo': 'COL50'}, {'styleText': '71 PURPLE', 'colorNo': 'COL71'}] 0 590.0 華夫格圓領T恤(長袖) 590.0 444514 3XL Y [new_product] LIST 女裝 女裝 華夫格圓領T恤(長袖) 444514 new_product u0000000011620
2 790.0 5.0 N /hmall/test/u0000000011478/main/first/1000/1.jpg XS Y Y 女裝 black [{'styleText': '01 OFF WHITE', 'colorNo': 'COL01'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}, {'styleText': '41 YELLOW', 'colorNo': 'COL41'}, {'styleText': '62 BLUE', 'colorNo': 'COL62'}] 1 790.0 圓領短版休閒上衣(長袖) 790.0 444533 3XL Y [new_product] LIST 女裝 女裝 圓領短版休閒上衣(長袖) 444533 new_product u0000000011478
3 590.0 0.0 N /hmall/test/u0000000011503/main/first/1000/1.jpg XS Y Y 女裝 black [{'styleText': '01 OFF WHITE', 'colorNo': 'COL01'}, {'styleText': '42 YELLOW', 'colorNo': 'COL42'}, {'styleText': '65 BLUE', 'colorNo': 'COL65'}, {'styleText': '69 NAVY', 'colorNo': 'COL69'}] 0 590.0 女裝條紋寬版T恤(長袖) 590.0 444516 3XL Y [new_product] LIST 女裝 女裝 女裝條紋寬版T恤(長袖) 444516 new_product u0000000011503
4 790.0 4.6316 N NaN S Y Y 女裝 red [{'styleText': '00 WHITE', 'colorNo': 'COL00'}, {'styleText': '08 DARK GRAY', 'colorNo': 'COL08'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}, {'styleText': '30 NATURAL', 'colorNo': 'COL30'}, {'styleText': '58 DARK GREEN', 'colorNo': 'COL58'}, {'styleText': '68 BLUE', 'colorNo': 'COL68'}] 19 390.0 HEATTECH棉質 中高領T恤 (長袖) 390.0 441587 3XL Y [concessional_rate] LIST 女裝 女裝 HEATTECH棉質 中高領T恤 (長袖) 441587 concessional_rate u0000000010558
5 590.0 4.3636 N /hmall/test/u0000000010146/main/first/1000/1.jpg XS Y Y 女裝 black [{'styleText': '01 OFF WHITE', 'colorNo': 'COL01'}, {'styleText': '17 RED', 'colorNo': 'COL17'}, {'styleText': '34 BROWN', 'colorNo': 'COL34'}, {'styleText': '69 NAVY', 'colorNo': 'COL69'}] 11 590.0 條紋寬版T恤 (長袖) 590.0 442793 3XL Y [EXTENDED SIZE] LIST 女裝 女裝 條紋寬版T恤 (長袖) 442793 EXTENDED SIZE u0000000010146
6 4990.0 3.3333 N /hmall/test/u0000000011297/main/first/1000/1.jpg XS Y Y 女裝 black [{'styleText': '08 DARK GRAY', 'colorNo': 'COL08'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}] 3 4990.0 J 絲質領結上衣 (長袖) 4990.0 446290 XXL Y [EXTENDED SIZE] LIST 女裝 女裝 J 絲質領結上衣 (長袖) 446290 EXTENDED SIZE u0000000011297
7 4990.0 5.0 N /hmall/test/u0000000011257/main/first/1000/1.jpg XS Y Y 女裝 black [{'styleText': '34 BROWN', 'colorNo': 'COL34'}, {'styleText': '76 PURPLE', 'colorNo': 'COL76'}] 1 4990.0 J 絲質上衣 (長袖) 4990.0 446577 XXL Y [EXTENDED SIZE] LIST 女裝 女裝 J 絲質上衣 (長袖) 446577 EXTENDED SIZE u0000000011257
8 990.0 4.6667 N /hmall/test/u0000000011030/main/first/1000/1.jpg XS Y Y 女裝 red [{'styleText': '01 OFF WHITE', 'colorNo': 'COL01'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}, {'styleText': '31 BEIGE', 'colorNo': 'COL31'}, {'styleText': '57 OLIVE', 'colorNo': 'COL57'}] 15 490.0 White Mountaineering 刷毛寬版中高領上衣 (長袖) (品牌好友穿著款) 490.0 443226 3XL Y [concessional_rate] LIST 女裝 女裝 White Mountaineering 刷毛寬版中高領上衣 (長袖) (品牌好友穿著款) 443226 concessional_rate u0000000011030
9 590.0 4.963 N NaN XS Y Y 女裝 black [{'styleText': '01 OFF WHITE', 'colorNo': 'COL01'}, {'styleText': '03 GRAY', 'colorNo': 'COL03'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}, {'styleText': '31 BEIGE', 'colorNo': 'COL31'}, {'styleText': '32 BEIGE', 'colorNo': 'COL32'}, {'styleText': '65 BLUE', 'colorNo': 'COL65'}, {'styleText': '78 PURPLE', 'colorNo': 'COL78'}] 27 590.0 柔軟針織圓領 泡芙袖T恤 (長袖) 590.0 439142 3XL Y [EXTENDED SIZE] LIST 女裝 女裝 柔軟針織圓領 泡芙袖T恤 (長袖) 439142 EXTENDED SIZE u0000000010990
10 590.0 4.6 N /hmall/test/u0000000011073/main/first/1000/1.jpg XS Y Y 女裝 black [{'styleText': '01 OFF WHITE', 'colorNo': 'COL01'}, {'styleText': '08 DARK GRAY', 'colorNo': 'COL08'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}, {'styleText': '12 PINK', 'colorNo': 'COL12'}, {'styleText': '68 BLUE', 'colorNo': 'COL68'}, {'styleText': '76 PURPLE', 'colorNo': 'COL76'}] 5 590.0 HEATTECH 彈性刷毛圓領T恤 (長袖) 590.0 440267 3XL Y [EXTENDED SIZE] LIST 女裝 女裝 HEATTECH 彈性刷毛圓領T恤 (長袖) 440267 EXTENDED SIZE u0000000011073
11 590.0 4.4 N /hmall/test/u0000000011100/main/first/1000/1.jpg XS Y Y 女裝 black [{'styleText': '01 OFF WHITE', 'colorNo': 'COL01'}, {'styleText': '07 GRAY', 'colorNo': 'COL07'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}, {'styleText': '12 PINK', 'colorNo': 'COL12'}, {'styleText': '31 BEIGE', 'colorNo': 'COL31'}, {'styleText': '34 BROWN', 'colorNo': 'COL34'}, {'styleText': '66 BLUE', 'colorNo': 'COL66'}, {'styleText': '69 NAVY', 'colorNo': 'COL69'}] 5 590.0 HEATTECH 刷毛高領T恤 (長袖) 590.0 442063 3XL Y [EXTENDED SIZE] LIST 女裝 女裝 HEATTECH 刷毛高領T恤 (長袖) 442063 EXTENDED SIZE u0000000011100
12 590.0 4.5909 N /hmall/test/u0000000009255/main/first/1000/1.jpg XS Y Y 女裝 black [{'styleText': '01 OFF WHITE', 'colorNo': 'COL01'}, {'styleText': '08 DARK GRAY', 'colorNo': 'COL08'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}, {'styleText': '30 NATURAL', 'colorNo': 'COL30'}, {'styleText': '32 BEIGE', 'colorNo': 'COL32'}] 22 590.0 華夫格圓領T恤 (長袖) 590.0 439067 3XL Y [EXTENDED SIZE] LIST 女裝 女裝 華夫格圓領T恤 (長袖) 439067 EXTENDED SIZE u0000000009255
13 390.0 4.75 N NaN XS Y Y 女裝 black [{'styleText': '00 WHITE', 'colorNo': 'COL00'}, {'styleText': '02 LIGHT GRAY', 'colorNo': 'COL02'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}, {'styleText': '29 DARK ORANGE', 'colorNo': 'COL29'}, {'styleText': '47 YELLOW', 'colorNo': 'COL47'}, {'styleText': '69 NAVY', 'colorNo': 'COL69'}] 12 390.0 彈性柔滑棉圓領T恤 (長袖) 390.0 439062 3XL Y [EXTENDED SIZE] LIST 女裝 女裝 彈性柔滑棉圓領T恤 (長袖) 439062 EXTENDED SIZE u0000000009304
14 390.0 5.0 N /hmall/test/u0000000010162/main/first/1000/1.jpg S Y Y 女裝 black [{'styleText': '00 WHITE', 'colorNo': 'COL00'}, {'styleText': '07 GRAY', 'colorNo': 'COL07'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}, {'styleText': '17 RED', 'colorNo': 'COL17'}] 1 390.0 彈性柔滑棉 高領T恤 (長袖) 390.0 439063 XXL Y [EXTENDED SIZE] LIST 女裝 女裝 彈性柔滑棉 高領T恤 (長袖) 439063 EXTENDED SIZE u0000000010162
15 590.0 4.8095 N NaN L Y Y 女裝 red [{'styleText': '00 WHITE', 'colorNo': 'COL00'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}] 21 390.0 柔滑棉 設計下擺T恤 (長袖) 390.0 439065 L Y [concessional_rate] LIST 女裝 女裝 柔滑棉 設計下擺T恤 (長袖) 439065 concessional_rate u0000000009305
16 590.0 4.375 N NaN XS Y Y 女裝 red [{'styleText': '01 OFF WHITE', 'colorNo': 'COL01'}, {'styleText': '10 PINK', 'colorNo': 'COL10'}] 16 190.0 棉質羅紋短版開襟外套 (長袖) (品牌好友穿著款) 190.0 446249 3XL Y [concessional_rate] LIST 女裝 女裝 棉質羅紋短版開襟外套 (長袖) (品牌好友穿著款) 446249 concessional_rate u0000000009409
17 390.0 4.1111 N /hmall/test/u0000000009878/main/first/1000/1.jpg XS N N 女裝 red [{'styleText': '00 WHITE', 'colorNo': 'COL00'}, {'styleText': '08 DARK GRAY', 'colorNo': 'COL08'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}, {'styleText': '12 PINK', 'colorNo': 'COL12'}, {'styleText': '31 BEIGE', 'colorNo': 'COL31'}, {'styleText': '66 BLUE', 'colorNo': 'COL66'}] 9 190.0 羅紋方領T恤 (長袖) 190.0 445628 3XL Y [concessional_rate] LIST 女裝 女裝 羅紋方領T恤 (長袖) 445628 concessional_rate u0000000009878
18 790.0 4.6667 N /hmall/test/u0000000008087/main/first/1000/1.jpg XS N N 女裝 red [{'styleText': '01 OFF WHITE', 'colorNo': 'COL01'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}, {'styleText': '18 WINE', 'colorNo': 'COL18'}] 6 390.0 喬其紗印花荷葉邊上衣(長袖) 390.0 443345 3XL Y [concessional_rate] DELIST 女裝 女裝 喬其紗印花荷葉邊上衣(長袖) 443345 concessional_rate u0000000008087
19 790.0 4.3158 N /hmall/test/u0000000008124/main/first/1000/1.jpg XS N N 女裝 red [{'styleText': '01 OFF WHITE', 'colorNo': 'COL01'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}] 19 390.0 印花V領澎袖上衣 (長袖) (品牌好友穿著款) 390.0 439736 3XL Y [concessional_rate] LIST 女裝 女裝 印花V領澎袖上衣 (長袖) (品牌好友穿著款) 439736 concessional_rate u0000000008124
20 790.0 3.8 N /hmall/test/u0000000010858/main/first/1000/1.jpg XS Y Y 女裝 red [{'styleText': '00 WHITE', 'colorNo': 'COL00'}, {'styleText': '04 GRAY', 'colorNo': 'COL04'}, {'styleText': '09 BLACK', 'colorNo': 'COL09'}, {'styleText': '11 PINK', 'colorNo': 'COL11'}, {'styleText': '55 GREEN', 'colorNo': 'COL55'}] 5 390.0 Dry-EX 圓領T恤 (長袖) 390.0 442388 3XL Y [concessional_rate] LIST 女裝 女裝 Dry-EX 圓領T恤 (長袖) 442388 concessional_rate u0000000010858