I can't access elements on ESPN Fantasy when using BS4 or Selenium-CodePudding

sample URL: https://fantasy.espn.com/basketball/team?leagueId=454981630&seasonId=2023&teamId=10

So what I want to do is basically grab the names of players in someone's fantasy team. All the names are located under and each player is listed under "Table__TR Table__TR--lg Table__odd".

I tried using BS4 and the soup.find_all(class_=(Table__TBODY)) but it kept returning none. (My guess would be the site's javascript messed with bs4).

I then switched over to selenium and ran the following to no avail.

# pull player from espn team


from selenium import webdriver
from selenium.webdriver.common.by import By
PATH = "C:\ChromeDrive\chromedriver.exe"
driver = webdriver.Chrome(PATH)

driver.get("https://fantasy.espn.com/basketball/team?leagueId=454981630&seasonId=2023&teamId=10
")
print(driver.title)

search = driver.find_element(By.CSS_SELECTOR, 'Table__TR Table__TR--lg Table__odd')
print(search)

However all I got was an error saying the element didn't exist. What's going on? I know the element exists because I'm looking right at it. At first I thought it was because the league was private and selenium couldn't actually access the site the way I saw it but nope, league is public and viewable to anyone.

I would greatly appreciate it if someone could explain why I can't access the element and how I could improve the code. Thank you.

CodePudding user response：

If you want to use selenium, here is one way to get those names (and only names. You will need some additional imports as well:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

[..]
wait = WebDriverWait(driver, 20)

driver.get(url)

players = wait.until(EC.presence_of_all_elements_located((By.XPATH, '//div[contains(@class, "player-column__athlete")]')))
print([x.find_element(By.TAG_NAME, 'a').text for x in players])

Result in terminal:

['Luka Doncic', 'Bradley Beal', 'Paul George', 'Draymond Green', 'Wendell Carter Jr.', 'Chris Paul', 'Jerami Grant', 'CJ McCollum', 'Malcolm Brogdon', 'Russell Westbrook', 'John Wall', 'Aaron Gordon', 'Bojan Bogdanovic']

CodePudding user response：

In alternative you could also use request only while extracting the information from an api, that also will give you a lot of other information in JSON structure.

Example

import requests

url = 'https://fantasy.espn.com/apis/v3/games/fba/seasons/2023/segments/0/leagues/454981630?forTeamId=10&scoringPeriodId=1&view=mRoster'
r = requests.get(url, headers = {"User-Agent": "Mozilla/5.0"})
for e in r.json()['teams'][0]['roster']['entries']:
    print(e['playerPoolEntry']['player']['fullName'])

Output

Luka Doncic
Paul George
Bradley Beal
Chris Paul
CJ McCollum
Malcolm Brogdon
Russell Westbrook
Draymond Green
Wendell Carter Jr.
Jerami Grant
John Wall
Aaron Gordon
Bojan Bogdanovic