Home > Mobile >  How can I scrape the position of the players in Selenium?
How can I scrape the position of the players in Selenium?

Time:04-30

If you visit this site, https://www.premierleague.com/players, you will be able to see a list of players with their positions and Nationality. I am only required to scrape the position but I am unable to

playerss = driver.find_elements(By.XPATH, '//*[@id="mainContent"]/div[2]/div[1]/div/div/table/tbody/tr')

for player in playerss:
    position = player.find_element(By.XPATH,'//[@id="mainContent"]/div[2]/div[1]/div/div/table/tbody/tr/td[2]')
    print(position)

This is what I have but it is not working. If anyone knows how to help, I will appreciate it.

Thanks!

CodePudding user response:

This is how I was able to get the list of positions:

...
driver.get('https://www.premierleague.com/players')
positions = driver.find_elements(By.CSS_SELECTOR, 
                                 'table tbody tr td:nth-child(2)')

for pos in positions:
    print(pos.text)

if you have to use xpath:

positions = driver.find_elements(By.XPATH, '//table/tbody/tr/td[2]')

CodePudding user response:

The position of the players are within:

<td >Midfielder</td>

where the <td> is having class as hide-s


Solution

To print the positions you need to induce WebDriverWait for the presence_of_element_located() and you can use either of the following locator strategies:

  • Using text attribute:

    print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table tbody tr td:nth-of-type(2)")))])
    
  • Using innerHTML:

    print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table tbody tr td:nth-of-type(2)")))])
    
  • Using innerText:

    print([my_elem.get_attribute("innerText") for my_elem in WebDriverWait(driver, 20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "table tbody tr td:nth-of-type(2)")))])
    
  • Console Output:

    ['Midfielder', 'Defender', 'Defender', 'Forward', 'Forward', 'Forward', 'Defender', 'Goalkeeper', 'Forward', 'Defender', 'Defender', 'Defender', 'Defender', 'Defender', 'Midfielder', 'Defender', 'Defender', 'Midfielder', 'Defender', 'Defender', 'Defender', 'Midfielder', 'Goalkeeper', 'Midfielder', 'Midfielder', 'Midfielder', 'Defender', 'Midfielder', 'Forward', 'Forward']
    
  • Related