How to get a specific network item and header value using selenium?-CodePudding

I am trying to get the price from this website, however my code does not return the price. I am wondering if I can get a specific network item header values.

For example network with name 759454 and header where it contains fingerprint? With that I can scrape the whole thing. Currently using class name does not return the price

import requests
import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By


url='https://alfagift.id/p/elmondo-jas-hujan-ceria-setelan-jaket-celana-759454'

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(ChromeDriverManager().install(),options=options)
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
print(driver.execute_script("return navigator.userAgent;"))

driver.get(url)

price= driver.find_elements(By.CLASS_NAME,"text-xlg fw7 text-primary mb-1")
print(price)

for x in price:
    print(x.text)

CodePudding user response：

By.CLASS_NAME() only accepts single class name not multiple class names. Instead use By.CSS_SELECTOR().

Use WebDriverWait() and wait for presence_of_element_located() and use textContent attribute to get the value. Since element is hidden in the DOM tree.

You need to import below libraries as well

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By

Code

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
driver.get('https://alfagift.id/p/elmondo-jas-hujan-ceria-setelan-jaket-celana-759454')
price=WebDriverWait(driver,10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "p.text-primary.mb-1"))).get_attribute("textContent")
print(price)

Output: