I'm trying to scrape player data from college football roster sites. I am primarily interested in getting the player image, weight, and name. I have already been able to extract the weight and name but am struggling on extracting the image using selenium. This is my code so far.
driver = webdriver.Chrome("C:/Users/<my_user>/Downloads/chromedriver.exe")
driver.get(school["url"])
all_names = driver.find_elements(by=By.CLASS_NAME, value='sidearm-roster-player-name')
all_weights = driver.find_elements(by=By.CLASS_NAME, value='sidearm-roster-player-weight')
all_imgs = driver.find_elements(by=By.CLASS_NAME, value='sidearm-roster-player-image')
This is an example of what would be passed in as school[url]
. Many colleges use this format.
https://rolltide.com/sports/football/roster
Each player on this site has the following html element.
<div >
<a data-bind="click: function() { return true; }, clickBubble: false" href="/sports/football/roster/jeremiah-alexander/8141" aria-label="Jeremiah Alexander - View Full Bio" title="View Full Bio">
<img data-src="https://d1a8hwz3c6qyrc.cloudfront.net/images/2022/3/1/Alexander_Jeremiah.jpg?width=80" alt="Jeremiah Alexander" src="https://d1a8hwz3c6qyrc.cloudfront.net/images/2022/3/1/Alexander_Jeremiah.jpg?width=80">
</a>
</div>
The issue I am running into is that each webElement
in all_imgs
does not seem to have an attribute such called 'img' or any attribute I can see that represents the image link located within the element. How can I get the link of all the images of the players on this page?
CodePudding user response:
Actually... you can get all the data without selenium
. Everything is in the source HTML.
Here's how to do this:
import json
import requests
from bs4 import BeautifulSoup
from tabulate import tabulate
url = "https://rolltide.com/sports/football/roster"
soup = BeautifulSoup(requests.get(url).text, "lxml")
height_weight = [
" ".join(i.getText(strip=True, separator=" ").split()) for i in
soup.select(".sidearm-roster-default-template .sidearm-roster-player-position")
]
images_and_names = [
[
player['name'],
player['url'],
player['image']['url']
] for player in
json.loads(soup.find("script", {"type": "application/ld json"}).string)["item"]
]
player_data = [[i, *j] for i, j in zip(height_weight, images_and_names)]
print(tabulate(player_data, headers=["Body Info", "Name", "Full Bio", "Image"]))
This should output:
Body Info Name Full Bio Image
------------------ -------------------- ------------------------------------------ -------------------------------------------------------------
LB 6'2" 258 lbs Jeremiah Alexander http://rolltide.com/roster.aspx?rp_id=8141 http://rolltide.com/images/2022/3/1/Alexander_Jeremiah.jpg
PK 6'2" 188 lbs Chase Allen http://rolltide.com/roster.aspx?rp_id=8048 http://rolltide.com/images/2020/7/8/Allen_Chase.jpg
WR 5'9" 184 lbs Aaron Anderson http://rolltide.com/roster.aspx?rp_id=8142 http://rolltide.com/images/2022/3/1/Anderson_Aaron.jpg
LB 6'4" 243 lbs Will Anderson Jr. http://rolltide.com/roster.aspx?rp_id=8049 http://rolltide.com/images/2020/3/19/Anderson_Jr_Will_71.jpg
DB 6'0" 188 lbs Terrion Arnold http://rolltide.com/roster.aspx?rp_id=8050 http://rolltide.com/images/2021/8/6/Aronld_Terrion.jpg
DL 6'5" 305 lbs Anquin Barnes Jr. http://rolltide.com/roster.aspx?rp_id=8051 http://rolltide.com/images/2021/8/6/Barnes_Anquin.jpg
DB 6'1" 206 lbs Jordan Battle http://rolltide.com/roster.aspx?rp_id=8052 http://rolltide.com/images/2020/3/19/Battle_Jordan_71.jpg
RB 5'8" 180 lbs Jonathan Bennett http://rolltide.com/roster.aspx?rp_id=8053 http://rolltide.com/images/2020/11/2/Bennett_Jonathan_71.jpg
LB 6'2" 233 lbs Kendrick Blackshire http://rolltide.com/roster.aspx?rp_id=8054 http://rolltide.com/images/2021/8/6/Blackshire_Kendrick.jpg
WR 6'0" 176 lbs Bret Bolin http://rolltide.com/roster.aspx?rp_id=8055 http://rolltide.com/images/2020/7/8/Bolin_Bret.jpg
OL 6'5" 332 lbs Tyler Booker http://rolltide.com/roster.aspx?rp_id=8143 http://rolltide.com/images/2022/3/1/Booker_Tyler.jpg
OL 6'5" 293 lbs Tanner Bowles http://rolltide.com/roster.aspx?rp_id=8056 http://rolltide.com/images/2020/3/19/Bowles_Tanner_71.jpg
WR 5'11" 182 lbs Jacoby Boykins http://rolltide.com/roster.aspx?rp_id=8057 http://rolltide.com/images/2021/8/6/Boykins_Jacoby.jpg
DB 6'0" 193 lbs Brian Branch http://rolltide.com/roster.aspx?rp_id=8058 http://rolltide.com/images/2021/6/17/Branch_Brian.jpg
LB 6'3" 240 lbs Chris Braswell http://rolltide.com/roster.aspx?rp_id=8059 http://rolltide.com/images/2020/3/19/Braswell_Chris_71.jpg
OL 6'3" 282 lbs James Brockermeyer http://rolltide.com/roster.aspx?rp_id=8060 http://rolltide.com/images/2021/2/9/Brockermeyer_James_71.jpg
OL 6'5" 304 lbs Tommy Brockermeyer http://rolltide.com/roster.aspx?rp_id=8061 http://rolltide.com/images/2021/2/9/Brockermeyer_Tommy.jpg
WR 6'2" 196 lbs Ja'Corey Brooks http://rolltide.com/roster.aspx?rp_id=8062 http://rolltide.com/images/2021/2/9/Brooks_Ja_Corey.jpg
TE 6'5" 238 lbs Elijah Brown http://rolltide.com/roster.aspx?rp_id=8144 http://rolltide.com/images/2022/3/1/Brown_Elijah.jpg
WR 6'3" 175 lbs J'Kolbe Bulock http://rolltide.com/roster.aspx?rp_id=8145 http://rolltide.com/images/2022/3/1/Bulock_J_Kolbe.jpg
P 6'6" 211 lbs James Burnip http://rolltide.com/roster.aspx?rp_id=8063 http://rolltide.com/images/2021/8/6/Burnip_James.jpg
DL 6'3" 309 lbs Jamil Burroughs http://rolltide.com/roster.aspx?rp_id=8064 http://rolltide.com/images/2020/7/8/Burroughs_Jamil.jpg
WR 6'0" 200 lbs Jermaine Burton http://rolltide.com/roster.aspx?rp_id=8146 http://rolltide.com/images/2022/3/1/Burton_Jermaine.jpg
LB 6'3" 225 lbs Jihaad Campbell http://rolltide.com/roster.aspx?rp_id=8147 http://rolltide.com/images/2022/3/1/Campbell_Jihaad.jpg
OL 6'4" 305 lbs Javion Cohen http://rolltide.com/roster.aspx?rp_id=8065 http://rolltide.com/images/2020/7/8/Cohen_Javion.jpg
RB 5'11" 210 lbs Elijah Crockett http://rolltide.com/roster.aspx?rp_id=8067 http://rolltide.com/images/2021/8/20/Crockett_Elijah.jpg
OL 6'3" 305 lbs Darrian Dalcourt http://rolltide.com/roster.aspx?rp_id=8068 http://rolltide.com/images/2020/3/19/Dalcourt_Darrian_53.jpg
DL 6'3" 300 lbs DJ Dale http://rolltide.com/roster.aspx?rp_id=8069 http://rolltide.com/images/2021/8/20/Dale_DJ.jpg
WR 5'10" 177 lbs JoJo Earle http://rolltide.com/roster.aspx?rp_id=8070 http://rolltide.com/images/2021/8/6/Earle_JoJo.jpg
DL 6'5" 292 lbs Justin Eboigbe http://rolltide.com/roster.aspx?rp_id=8071 http://rolltide.com/images/2020/8/17/Eboigbe_Justin.jpg
OL 6'3" 307 lbs Emil Ekiyor Jr. http://rolltide.com/roster.aspx?rp_id=8073 http://rolltide.com/images/2020/8/17/Ekiyor_Jr_Emil.jpg
TE 6'0" 220 lbs Robert Ellis http://rolltide.com/roster.aspx?rp_id=8074 http://rolltide.com/images/2020/11/2/Ellis_Robert.jpg
DB 6'2" 185 lbs Tre'Quon Fegans http://rolltide.com/roster.aspx?rp_id=8148 http://rolltide.com/images/2022/3/1/Fegans_Tre_Quon.jpg
OL 6'4" 300 lbs Terrence Ferguson II http://rolltide.com/roster.aspx?rp_id=8075 http://rolltide.com/images/2021/2/9/Ferguson_Terrence.jpg
LB 6'0" 212 lbs Kyle Flood Jr. http://rolltide.com/roster.aspx?rp_id=8076 http://rolltide.com/images/2022/3/1/Flood_Jr_Kyle.jpg
OL 6'6" 333 lbs Damieon George Jr. http://rolltide.com/roster.aspx?rp_id=8077 http://rolltide.com/images/2020/7/8/George_Jr_Damieon.jpg
RB 5'11" 200 lbs Jahmyr Gibbs http://rolltide.com/roster.aspx?rp_id=8149 http://rolltide.com/images/2022/3/1/Gibbs_Jahmyr.jpg
DL 6'4" 288 lbs Monkell Goodwine http://rolltide.com/roster.aspx?rp_id=8078 http://rolltide.com/images/2021/2/9/Goodwine_Monkell.jpg
WR 6'3" 202 lbs Agiye Hall http://rolltide.com/roster.aspx?rp_id=8079 http://rolltide.com/images/2021/2/9/Hall_Agiye.jpg
DB 6'1" 208 lbs DeMarcco Hellams http://rolltide.com/roster.aspx?rp_id=8080 http://rolltide.com/images/2020/3/19/Hellams_DeMarcco_58.jpg
SN 6'2" 245 lbs Kneeland Hibbett http://rolltide.com/roster.aspx?rp_id=8081 http://rolltide.com/images/2021/8/6/Hibbett_Kneeland.jpg
WR 6'3" 214 lbs Traeshon Holden http://rolltide.com/roster.aspx?rp_id=8082 http://rolltide.com/images/2020/3/19/Holden_Traeshon_53.jpg
DL 6'4" 295 lbs Braylen Ingraham http://rolltide.com/roster.aspx?rp_id=8084 http://rolltide.com/images/2020/3/19/Ingraham_Braylen_53.jpg
LB 6'1" 235 lbs Ian Jackson http://rolltide.com/roster.aspx?rp_id=8085 http://rolltide.com/images/2021/2/9/Jackson_Ian.jpg
DB 6'3" 198 lbs Khyree Jackson http://rolltide.com/roster.aspx?rp_id=8086 http://rolltide.com/images/2021/8/6/Jackson_Khyree.jpg
QB 6'2" 186 lbs Blake Jarrett http://rolltide.com/roster.aspx?rp_id=8150 http://rolltide.com/images/2022/3/1/Jarrett_Blake.jpg
LB 6'5" 230 lbs Christian Johnson http://rolltide.com/roster.aspx?rp_id=8087 http://rolltide.com/images/2021/8/16/Johnson_Christian.jpg
WR 6'0" 190 lbs Thaiu Jones-Bell http://rolltide.com/roster.aspx?rp_id=8089 http://rolltide.com/images/2020/3/19/Jones_Bell_Thaiu_71.jpg
DL 6'2" 343 lbs Tim Keenan III http://rolltide.com/roster.aspx?rp_id=8090 http://rolltide.com/images/2021/8/6/Keenan_III_Tim.jpg
LB 6'3" 220 lbs Demouy Kennedy http://rolltide.com/roster.aspx?rp_id=8091 http://rolltide.com/images/2020/8/20/Kennedy_Demouy.jpg
OL 6'7" 322 lbs Amari Kight http://rolltide.com/roster.aspx?rp_id=8092 http://rolltide.com/images/2020/3/19/Kight_Amari_81.jpg
LB 6'4" 231 lbs Keanu Koht http://rolltide.com/roster.aspx?rp_id=8093 http://rolltide.com/images/2021/2/9/Koht_Keanu.jpg
WR 6'2" 192 lbs Grant Krieger http://rolltide.com/roster.aspx?rp_id=8094 http://rolltide.com/images/2020/3/19/Krieger_Grant_30.jpg
DB 6'1" 170 lbs Brylan Lanier http://rolltide.com/roster.aspx?rp_id=8095 http://rolltide.com/images/2021/8/20/Lanier_Brylan.jpg
DL 6'3" 278 lbs Jah-Marien Latham http://rolltide.com/roster.aspx?rp_id=8096 http://rolltide.com/images/2020/7/8/Latham_Jah_Marien.jpg
OL 6'6" 326 lbs JC Latham http://rolltide.com/roster.aspx?rp_id=8097 http://rolltide.com/images/2021/2/9/Latham_JC.jpg
TE 6'5" 244 lbs Cameron Latu http://rolltide.com/roster.aspx?rp_id=8098 http://rolltide.com/images/2020/3/19/Latu_Cameron_53.jpg
WR 5'11" 193 lbs Kendrick Law http://rolltide.com/roster.aspx?rp_id=8151 http://rolltide.com/images/2022/3/1/Law_Kendrick.jpg
LB 6'2" 225 lbs Deontae Lawson http://rolltide.com/roster.aspx?rp_id=8099 http://rolltide.com/images/2021/2/9/Lawson_Deontae.jpg
WR 5'10" 175 lbs Christian Leary http://rolltide.com/roster.aspx?rp_id=8100 http://rolltide.com/images/2021/2/9/Leary_Christian.jpg
P 6'2" 207 lbs Jack Martin http://rolltide.com/roster.aspx?rp_id=8102 http://rolltide.com/images/2021/8/6/Martin_Jack.jpg
DB 6'1" 143 lbs Jacobi McBride http://rolltide.com/roster.aspx?rp_id=8179 http://rolltide.com/images/2021/3/3/McBride_Jacobi.jpg
RB 5'11" 212 lbs Jase McClellan http://rolltide.com/roster.aspx?rp_id=8103 http://rolltide.com/images/2020/3/19/McClellan_Jase_58.jpg
DB 6'1" 188 lbs Kool-Aid McKinstry http://rolltide.com/roster.aspx?rp_id=8104 http://rolltide.com/images/2021/2/9/McKinstry_Ga_Quincy.jpg
OL 6'4" 295 lbs Seth McLaughlin http://rolltide.com/roster.aspx?rp_id=8105 http://rolltide.com/images/2020/5/1/McLaughlin_Seth.jpg
RB 5'10" 201 lbs Jamarion Miller http://rolltide.com/roster.aspx?rp_id=8152 http://rolltide.com/images/2022/3/1/Miller_Jamarion.jpg
QB 6'2" 212 lbs Jalen Milroe http://rolltide.com/roster.aspx?rp_id=8106 http://rolltide.com/images/2021/2/9/Milroe_Jalen.jpg
LB 6'2" 225 lbs Jaylen Moody http://rolltide.com/roster.aspx?rp_id=8107 http://rolltide.com/images/2020/3/19/Moody_Jaylen_53.jpg
DB 6'0" 190 lbs Malachi Moore http://rolltide.com/roster.aspx?rp_id=8108 http://rolltide.com/images/2020/7/8/Moore_Malachi.jpg
LB 6'2" 224 lbs Shawn Murphy http://rolltide.com/roster.aspx?rp_id=8153 http://rolltide.com/images/2022/3/1/Murphy_Shawn.jpg
DL 6'5" 370 lbs Jaheim Oatis http://rolltide.com/roster.aspx?rp_id=8154 http://rolltide.com/images/2022/3/1/Oatis_Jaheim.jpg
TE 6'4" 258 lbs Robbie Ouzts http://rolltide.com/roster.aspx?rp_id=8109 http://rolltide.com/images/2021/2/9/Ouzts_Robbie.jpg
DL 6'4" 303 lbs Damon Payne Jr. http://rolltide.com/roster.aspx?rp_id=8111 http://rolltide.com/images/2021/8/6/Payne_Damon.jpg
DL 6'2" 264 lbs Khurtiss Perry http://rolltide.com/roster.aspx?rp_id=8155 http://rolltide.com/images/2022/3/1/Perry_Khurtiss.jpg
TE 6'6" 232 lbs Jax Porter http://rolltide.com/roster.aspx?rp_id=8156 http://rolltide.com/images/2022/3/1/Porter_Jax_71.jpg
DB 6'0" 175 lbs Blake Pugh http://rolltide.com/roster.aspx?rp_id=8112 http://rolltide.com/images/2021/8/20/Pugh_Blake.jpg
SN 6'5" 273 lbs Gabe Pugh http://rolltide.com/roster.aspx?rp_id=8113 http://rolltide.com/images/2021/8/16/Pugh_Gabe.jpg
DL 6'1" 236 lbs Chase Quigley http://rolltide.com/roster.aspx?rp_id=8114 http://rolltide.com/images/2021/8/6/Quigley_Chase.jpg
TE/OL 6'4" 298 lbs Kendall Randolph http://rolltide.com/roster.aspx?rp_id=8115 http://rolltide.com/images/2020/3/19/Randolph_Kendall_29.jpg
PK 6'1" 190 lbs Will Reichard http://rolltide.com/roster.aspx?rp_id=8116 http://rolltide.com/images/2020/8/17/Reichard_Will.jpg
DB 6'2" 190 lbs Eli Ricks http://rolltide.com/roster.aspx?rp_id=8157 http://rolltide.com/images/2022/3/1/Ricks_Eli.jpg
OL 6'5" 302 lbs Jaeden Roberts http://rolltide.com/roster.aspx?rp_id=8117 http://rolltide.com/images/2021/8/6/Roberts_Jaeden.jpg
DB 6'2" 197 lbs Jahquez Robinson http://rolltide.com/roster.aspx?rp_id=8118 http://rolltide.com/images/2020/3/19/Robinson_Jahquez_30.jpg
LB 6'5" 224 lbs Quandarrius Robinson http://rolltide.com/roster.aspx?rp_id=8119 http://rolltide.com/images/2020/7/8/Robinson_Quandarrius.jpg
OL 6'5" 285 lbs Jackson Roby http://rolltide.com/roster.aspx?rp_id=8120 http://rolltide.com/images/2020/3/19/Roby_Jackson_71.jpg
OL 6'3" 285 lbs Graham Roten http://rolltide.com/roster.aspx?rp_id=8121 http://rolltide.com/images/2021/8/6/Roten_Graham.jpg
RB 6'0" 214 lbs Trey Sanders http://rolltide.com/roster.aspx?rp_id=8122 http://rolltide.com/images/2020/8/17/Sanders_Trey.jpg
PK 6'0" 185 lbs Reid Schuback http://rolltide.com/roster.aspx?rp_id=8123 http://rolltide.com/images/2021/8/6/Schuback_Reid.jpg
OL 6'5" 308 lbs Dayne Shor http://rolltide.com/roster.aspx?rp_id=8158 http://rolltide.com/images/2022/3/1/Shor_Dayne.jpg
QB 6'2" 198 lbs Ty Simpson http://rolltide.com/roster.aspx?rp_id=8159 http://rolltide.com/images/2022/3/1/Simpson_Ty.jpg
TE 6'1" 232 lbs Charlie Skehan http://rolltide.com/roster.aspx?rp_id=8125 http://rolltide.com/images/2020/12/4/Skehan_Charlie.jpg
DB 6'0" 185 lbs DeVonta Smith http://rolltide.com/roster.aspx?rp_id=8126 http://rolltide.com/images/2021/8/6/Smith_DeVonta.jpg
LB 5'10" 210 lbs Jordan Smith http://rolltide.com/roster.aspx?rp_id=8127 http://rolltide.com/images/2021/8/16/Smith_Jordan.jpg
DL 6'4" 304 lbs Tim Smith http://rolltide.com/roster.aspx?rp_id=8128 http://rolltide.com/images/2020/7/8/Smith_Tim.jpg
DB 6'1" 211 lbs Kristian Story http://rolltide.com/roster.aspx?rp_id=8129 http://rolltide.com/images/2020/7/8/Story_Kristian.jpg
TE 6'5" 232 lbs Adam Thorsland http://rolltide.com/roster.aspx?rp_id=8130 http://rolltide.com/images/2021/8/6/Thorsland_Adam.jpg
LB 6'2" 228 lbs Henry To'oTo'o http://rolltide.com/roster.aspx?rp_id=8131 http://rolltide.com/images/2021/8/6/To_oto_o_Henry.jpg
LB 6'4" 240 lbs Dallas Turner http://rolltide.com/roster.aspx?rp_id=8132 http://rolltide.com/images/2021/8/6/Turner_Dallas.jpg
SN 5'9" 185 lbs Kade Wehby http://rolltide.com/roster.aspx?rp_id=8133 http://rolltide.com/images/2022/3/1/Wehby_Kade.jpg
LB 6'1" 222 lbs Bennett Whisenhunt http://rolltide.com/roster.aspx?rp_id=8134 http://rolltide.com/images/2020/12/4/Whisenhunt_Bennett.jpg
DB 6'2" 203 lbs Kaine Williams http://rolltide.com/roster.aspx?rp_id=8135 http://rolltide.com/images/2021/8/6/Williams_Kaine.jpg
RB 5'10" 212 lbs Roydell Williams http://rolltide.com/roster.aspx?rp_id=8136 http://rolltide.com/images/2020/3/19/Williams_Roydell_53.jpg
WR 5'10" 165 lbs Sam Willoughby http://rolltide.com/roster.aspx?rp_id=8137 http://rolltide.com/images/2021/8/16/Willoughby_Sam.jpg
DL 6'4" 307 lbs Stephon Wynn Jr. http://rolltide.com/roster.aspx?rp_id=8138 http://rolltide.com/images/2020/3/19/Wynn_Jr_Stephon_71.jpg
QB 6'0" 194 lbs Bryce Young http://rolltide.com/roster.aspx?rp_id=8139 http://rolltide.com/images/2020/3/19/Young_Bryce_71.jpg
DL 6'3" 292 lbs Byron Young http://rolltide.com/roster.aspx?rp_id=8140 http://rolltide.com/images/2020/3/19/Young_Byron_76.jpg
CodePudding user response:
To create a list with all the values of the src
attribute you can use list comprehension and you can use either of the following locator strategies:
Using CSS_SELECTOR:
driver.get('https://rolltide.com/sports/football/roster') print([my_elem.get_attribute("src") for my_elem in driver.find_elements(By.CSS_SELECTOR, "a[title='View Full Bio'] > img")])
Using XPATH:
driver.get('https://rolltide.com/sports/football/roster') print([my_elem.get_attribute("src") for my_elem in driver.find_elements(By.XPATH, "//a[@title='View Full Bio']/img")])
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC