I have a program that reads data from a website using selenium. In this case I want to get the prices of the perfumes.
for i in driver.find_elements_by_css_selector("spam.items_show_price_text"):
print(i.text)
When I run the program it prints for me 139 empty lines(139 is the number of the perfumes that are in the website).
I guess it has to do with the "span.items_show_price_text"
parameter of the driver.find_elements_by_css_selector function.
The whole code (there is no need to understand the while loop, it is basicly runs the program for all of the page, because it is loding when you scroll down)
from datetime import date
import gspread
from oauth2client.service_account import ServiceAccountCredentials
from selenium import webdriver
from selenium.webdriver import ChromeOptions
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
urlM = 'https://www.myperfume.co.il/155567-כל-המותגים-לגב' \
'ר?order=up_title&page=0'
urlF = 'https://www.myperfume.co.il/155569-כל-המותגים-לאי' \
'שה?order=up_title&page=0'
scope = ["https://spreadsheets.google.com/feeds", 'https://www.googleapis.com/auth/spreadsheets',
"https://www.googleapis.com/auth/drive.file", "https://www.googleapis.com/auth/drive"]
creds = ServiceAccountCredentials.from_json_keyfile_name("credentials.json", scope)
client = gspread.authorize(creds)
spreadsheet = client.open("Perfumes")
options = ChromeOptions()
options.headless = True
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
namesM = []
pricesM = []
namesMe = ' '
lenM = 0
num = 0
while len(namesMe) != 0:
urlM = urlM[:-1] str(int(urlM[-1]) 1)
# ---=MALE=---
driver.get(urlM)
# names
for i in driver.find_elements_by_css_selector("h3.title.text-center"):
lenM = 1
namesM.append(i.text)
# prices
for i in driver.find_elements_by_css_selector("spam.items_show_price_text"):
print(i.text)
print(i.text)
CodePudding user response:
As I see on that site, the locator for actual prices (after the price off) will be
'span.price'
So, instead of
for i in driver.find_elements_by_css_selector("spam.items_show_price_text"):
Try using
for i in driver.find_elements_by_css_selector("span.price"):
You should also add waits/ delays there.
Again, it is preferably to use Expected Conditions explicit waits for that.
And don't mess spam
with span
:)