How to scrape price from Udemy?-CodePudding

Note: I don't find a relevant worked solution in any other similar questions.

My attempt is below.

import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager

options = Options()

driver = webdriver.Chrome(ChromeDriverManager().install())


url = "https://www.udemy.com/courses/development/?p=1"
driver.get(url)
time.sleep(2)


#data = driver.find_element('//div[@]')
#data  = driver.find_element_by_xpath('//div[contains(@class, "price-text--price-part"]')
#data=driver.find_element_by_css_selector('div.udlite-sr-only[attrName="price-text--price-part"]')

print(data)

Not of them worked for me. So, is there a way to select elements by classes that contain a specific text?

In this example, the text to find is: "price-text--price-part"

CodePudding user response：

The first xpath doesn't highlight any element in the DOM.

The second xpath doesn't have a closing brackets for contains

//div[contains(@class, "price-text--price-part"]
should be
//div[contains(@class, "price-text--price-part")]

Try like below, it might work. (When I tried the website detected as a bot and price was not loaded)

driver.get("https://www.udemy.com/courses/development/?p=1")

options = driver.find_elements_by_xpath("//div[contains(@class,'course-list--container')]/div[contains(@class,'popper')]")

for opt in options:
    title = opt.find_element_by_xpath(".//div[contains(@class,'title')]").text # Use a dot in the xpath to find element within in an element.
    price = opt.find_element_by_xpath(".//div[contains(@class,'price-text--price-part')]/span[2]/span").text
    print(f"{title}: {price}")