I would like to scrape the data from a website "https://www.maxifoot.fr/classement-buteur-europe-annee-civile-2021.htm"

I tried to extract these data on Python but I couldn't make it. I would like to create a table on Python with thoses data and the same fields. Can someone try to help me with the script of the data extraction using pandas, beautifulsoup... ?

I already tried this :

import requests
from bs4 import BeautifulSoup
import pandas as pd
url = 'https://www.maxifoot.fr/classement-buteur-europe-annee-civile-2021.htm'
r = requests.get(url)

soup = BeautifulSoup(r.text, 'lxml')
print(soup.prettify())
for i in soup.find_all("tr"):
    print(i.find_all("td"))
    print ("")
    colonnes = ["Nom","Equipe","Buts","Matchs joués"]
    df = pd.DataFrame(columns = colonnes)
df

for i in soup.find_all("tr")[1:]:
    href = i.find_all("td")
    df = df.append({'Nom': href}, ignore_index=True)

print(df.head())

CodePudding user response：

There is an much simpler way to grab the data and put it in a dataframe - Use pandas.read_html and make your adjustments if needed with pandas.

df = pd.read_html('https://www.maxifoot.fr/classement-buteur-europe-annee-civile-2021.htm', match = 'min/but*')[1]

df['href'] = df["Joueur"].apply(lambda x: 'https://www.maxifoot.fr' soup.select_one(f'a:-soup-contains("{x}")')['href'])

If you like to solve it with BeautifulSoup, there are some adjustments you have to do:

...
data = []
for row in soup.select('.butd1 tr')[1:]:
    strings = list(row.stripped_strings)
    strings[3:5] = [''.join(strings[3:5])]
    strings[6:8] = [''.join(strings[6:8])]
    a = 'https://www.maxifoot.fr' row.a['href']
    strings.append(a)
    data.append(strings)
colonnes = ['Pos','Nom','Equipe','Buts','dontchamp.','Matchs joués','min/but*','href']
pd.DataFrame(data,columns = colonnes)

Output

...