Assistance with scraping in selenium-CodePudding

This question was asked in June (Assistance needed scraping a site with Selenium in Python) and I am using the code from then, but running into problems. I assume there has been a change to the PP site. Here is my updated code that seems to be running into an error at clicking the Fantasy Score portion. The goal of the project is just to scrape the fantasy scores from NBA players

from time import sleep
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import chromedriver_binary

# Assign PrizePicks URL

driver = webdriver.Chrome()
PATH = "C:\Program Files (x86)\chromedriver.exe"
############## PRIZEPICKS ################################################
#
url = "https://app.prizepicks.com/"

driver.get(url)

#this is to get rid of the pop-up box that shows after you Selenium opens the prizepicks page
driver.find_element(By.CLASS_NAME,"close").click()

#Selecting NBA
driver.find_element(By.XPATH, "//div[@class='name'][normalize-space()= 'NBA']").click()

#Clicking the Fantasy Score tab
driver.find_element(By.XPATH, "//div[@class='stat stat-active'][normalize-space()='Fantasy Score']").click()
driver.find_element(By.XPATH, "//div[@class='segment-selector-button']").click()
projections = WebDriverWait(driver, 20).until(
 EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".projection")))

#creating a list of the players names and fantasy scores to be collected
nbaPlayers = []

for projection in projections:

    names = projection.find_element_by_xpath('.//div[@]').text
    points= projection.find_element_by_xpath('.//div[@]').get_attribute('innerHTML')
    print(names, points)

    players = {
        'Name': names,
        'FantasyPoints': points,
        }

    nbaPlayers.append(players)

#Put the list in a dataframe
df = pd.DataFrame(nbaPlayers)
print(df)

driver.quit()

CodePudding user response：

For the newer Selenium version it is advised to use

driver.find_element(By.XPATH, "//*[text()='Fantasy Score']").click()

CodePudding user response：

You can use "find_element_by_xpath" like this.

driver.find_element_by_xpath("//*[text()='Fantasy Score']").click()