website scraping not showing full html elements-CodePudding

I'm new to the world of programming and I've been trying to scrape a web site https://www.boove.se/butiker/mc but its not showing the full HTML elements of the website.

my code:

   from selenium import webdriver
   from selenium.webdriver.chrome.service import Service

   chromedriver_path = "C:\chromedriver_win32\chromedriver.exe"
   driver = webdriver.Chrome(service=Service(chromedriver_path))
   url = "https://www.boove.se/butiker/mc"
   driver.get(url)
   content = driver.find_elements(by=By.CSS_SELECTOR, value= "h3 .address desktop")
   print(content)
   driver.close()

CodePudding user response：

There are two issues with your code:

It needs the correct css selector to find the elements

You have to wait until the elements are located

wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'h3.address.desktop')))

Example

from selenium import webdriver
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = 'https://www.boove.se/butiker/mc'

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
driver.get(url)

wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'h3.address.desktop')))

content = driver.find_elements(By.CSS_SELECTOR, 'h3.address.desktop')

for i in content:
    print(i.text)
driver.close()

Output

Odenskogsvägen 35, 83148 Östersund
STORGATAN 43, 77100 Ludvika
INÄLVSVÄGEN 1, 69152 Karlskoga
VÄNE-RYR, 46293 Vänersborg
Västbergavägen 24, 12630 Hägersten
TENNGATAN 15, 23435 Lomma
HÄGERSTENS ALLE 12, 12937 Hägersten
SÅGVÄGEN 11A, 18440 Åkersberga
Sollentuna, 19278 Rotebro
Kuskvägen 2, 19162 Sollentuna
Mölndalsvägen 25, 41263 GÖTEBORG
SALAVÄGEN 5, 74537 Enköping
Lundbygårdsgata 3, 72134 Västerås
...

CodePudding user response：

You can easily scrape data using the Python library BeautifulSoup which is easier to use.

Here is the biolerplate code,

from bs4 import BeautifulSoup
import requests
url = ""
req = requests.get(url)
soup = BeautifulSoup(req.text, "html.parser")
print(soup)