I am trying to scrape player's name and their ratings on this website: https://www.whoscored.com/Matches/1549539/LiveStatistics/England-Premier-League-2021-2022-Brentford-Arsenal.

After scraping then I put the data in a csv. But, it does not scrape consistently. I probably have to run the script more than once(2-5 times) to get it to scrape the data. This also happens when I try to scrape for other matches. For example, if I get the data from 3 matches, probably it will only scrape the first match and doesn't scrape the remaining data for other pages. Here is my code:

from bs4 import BeautifulSoup
from selenium import webdriver

match_link='https://www.whoscored.com/Matches/1549539/Live/England-Premier-League-2021-2022-Brentford-Arsenal.'

driver=webdriver.Chrome('C:\\Program Files (x86)\\chromedriver.exe')
driver.get(match_link)
soup=BeautifulSoup(driver.page_source,'html.parser')

Players_list=[]
Player_rating=[]

try:
            player_name=soup.select('a.player-link span.iconize.iconize-icon-left')
            player_rating=soup.select('td.rating')

            #print('------------getting player name and ratings-----------')
            for nme in player_name:
                #print(nme.text)
                Players_list.append(nme.text)
            for rat in player_rating:
                #print(rat.text)
                Player_rating.append(rat.text)
except:
            print('NO ELEMENT')

Players_list=pd.DataFrame(Players_list)
Player_rating=pd.DataFrame(Player_rating)

        
df=pd.concat([Players_list,Player_rating],axis=1)
df.to_csv('brentford-arsenal.csv')

It doesn't raise an error. It just returns an empty results (meaning the data wasn't scraped).

Empty DataFrame
Columns: []
Index: []

CodePudding user response：

You should add an expicit wait for the page to render.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = 'https://www.whoscored.com/Matches/1549539/LiveStatistics/England-Premier-League-2021-2022-Brentford-Arsenal'

driver = webdriver.Chrome('C:\\Program Files (x86)\\chromedriver.exe')
driver.get(url)

WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, 'player-table-statistics-body'))
    )

Also, if you always want to use the latest chrome driver, then the new web manager auto-detects when a new driver is ready and caches it. To install manager: pip install webdriver-manager

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

s = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=s)

# web driver goes to page
driver.get(url)

...

CodePudding user response：

According to your question, The minimal working solution is as folows:

Script

from bs4 import BeautifulSoup
from selenium import webdriver
import pandas as pd
import time

match_link = 'https://www.whoscored.com/Matches/1549539/LiveStatistics/England-Premier-League-2021-2022-Brentford-Arsenal'

driver = webdriver.Chrome('chromedriver')
driver.maximize_window()
time.sleep(8)

driver.get(match_link)
time.sleep(5)


p=[]
q=[]
soup = BeautifulSoup(driver.page_source, 'html.parser')
divs = soup.select('#statistics-table-home-summary a span.iconize.iconize-icon-left')
for div in divs:
    player = div.text
    p.append(player)
    #print(player)

rs = soup.select('#statistics-table-home-summary td.rating')
for r in rs:
    rating = r.text
    #print(rating)
    q.append(rating)



cols= ['player','rating']
df = pd.DataFrame(data=list(zip(p,q)),columns=cols)
print(df)
# df.to_csv('brentford-arsenal.csv',index=False)

Output:

               player   rating
0           David Raya   7.60
1        Ethan Pinnock   7.57
2      Kristoffer Ajer   6.71
3       Pontus Jansson   6.92
4          Sergi Canós   8.78
5           Rico Henry   6.79
6   Christian Nørgaard   7.67
7        Vitaly Janelt   6.74
8         Frank Onyeka   7.06
9         Bryan Mbeumo   7.15
10          Ivan Toney   7.33
11       Mads Bidstrup   6.21
12  Mads Bech Sørensen   6.11
13        Marcus Forss   5.98
14       Mads Roerslev      -
15         Yoane Wissa      -
16       Saman Ghoddos      -
17    Halil Dervisoglu      -
18       Charlie Goode      -
19   Patrik Gunnarsson      -