How to get any result while scraping this website with selenium?-CodePudding

I am scraping the website with this code but it does not give me any result.

import json
import requests
from bs4 import BeautifulSoup
#import pandas as pd
import time
from selenium import webdriver
driver = webdriver.Chrome('D:/chromedriver_win32/chromedriver.exe')

#suit = []
url = 'https://www.asos.com/men/t-shirts-vests/cat/?cid=7616'
#   time.sleep(2)

# print(url)
driver.get(url)
pageSource = driver.page_source
soup = BeautifulSoup(pageSource,'lxml')
print(soup)

CodePudding user response：

This is one way to get the info from that page (of course, you can get more info, per categories, etc):

import requests
from bs4 import BeautifulSoup

headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}

r = requests.get('https://www.asos.com/men/t-shirts-vests/cat/?cid=7616', headers=headers)

soup = BeautifulSoup(r.text, 'html.parser')
articles = soup.select('article[data-auto-id="productTile"]')
for art in articles:
    print(art.text)

Result:

ASOS DESIGN t-shirt with crew neck in white - WHITE£6.00
ASOS DESIGN t-shirt with crew neck in black - BLACK£6.00
COLLUSION white t-shirt£5.99
COLLUSION black t-shirt£5.99
adidas Originals 'Sports Resort' Club t-shirt in white with back graphics£30.00
ASOS DESIGN cotton blend muscle fit t-shirt with crew neck in black - BLACK£6.00
ASOS DESIGN t-shirt with crew neck in white£6.00
COLLUSION Unisex oversized t-shirt with sun print in charcoal£15.99
The North Face Simple Dome logo t-shirt in green Exclusive at ASOS£24.00
adidas Training Train 365 logo vest in black£22.00
ASOS DESIGN t-shirt in beige with front city print£12.00
Pull&Bear Join Life t-shirt in white£9.99
French Connection crew neck t-shirt in black£10.00
Nike Running Run Dri-FIT t-shirt in peach£17.95
adidas Originals Happy Earth large back print earth trefoil logo t-shirt in off white£30.00
ASOS DESIGN 3 pack oversized t-shirt in multi£28.00
Topman oversized t-shirt in black£12.00
Vans left chest logo t-shirt in teal£21.00
COLLUSION oversized t-shirt with banana print in white£14.99
[...]

You don't need Selenium for this job. Requests documentation: https://requests.readthedocs.io/en/latest/

Also, BeautifulSoup docs can be found at https://www.crummy.com/software/BeautifulSoup/bs4/doc/

CodePudding user response：

As mentioned by @platipus_on_fire it is not necessary to use selenium but if so may start from a more up to date version and check docs

Note: executable_path has been deprecated, please pass in a Service object

Example

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
url = 'https://www.asos.com/men/t-shirts-vests/cat/?cid=7616'

driver.get(url)

soup = BeautifulSoup(driver.page_source)
data = []

for e in soup.select('article'):
    data.append({
        'title': e.h2.text,
        'price': e.p.text,
        'url': e.a.get('href')
    })
data

Output

[{'title': 'ASOS DESIGN t-shirt with crew neck in white - WHITE',
  'price': '£6.00',
  'url': 'https://www.asos.com/asos-design/asos-design-t-shirt-with-crew-neck-in-white-white/prd/22190861?clr=white&colourWayId=60380327&cid=7616'},
 {'title': 'ASOS DESIGN t-shirt with crew neck in black - BLACK',
  'price': '£6.00',
  'url': 'https://www.asos.com/asos-design/asos-design-t-shirt-with-crew-neck-in-black-black/prd/22190857?clr=black&colourWayId=60380297&cid=7616'},
 {'title': 'COLLUSION white t-shirt',
  'price': '£5.99',
  'url': 'https://www.asos.com/collusion/collusion-white-t-shirt/prd/202692601?clr=white&colourWayId=202692607&cid=7616'},
 {'title': 'COLLUSION black t-shirt',
  'price': '£5.99',
  'url': 'https://www.asos.com/collusion/collusion-black-t-shirt/prd/202692520?clr=black&colourWayId=202692530&cid=7616'},
 {'title': "adidas Originals 'Sports Resort' Club t-shirt in white with back graphics",
  'price': '£30.00',
  'url': 'https://www.asos.com/adidas-originals/adidas-originals-sports-resort-club-t-shirt-in-white-with-back-graphics/prd/201829282?clr=white&colourWayId=201829292&cid=7616'},...]