Web scraping doesn't iterate over entire webpage-CodePudding

Im trying to scrape the information of all the player names and player rating from this website: https://www.fifaindex.com/players/?gender=0&league=1&order=desc

But i only get the information from the first player on the page.

The code im using:


from bs4 import BeautifulSoup
import requests

url = "https://www.fifaindex.com/players/?gender=0&league=1&order=desc"

page = requests.get(url)

soup = BeautifulSoup(page.content, 'html.parser')

results = soup.find_all('div', class_="responsive-table table-rounded")

for result in results:
    rating = result.find("span", class_="badge badge-dark rating r3").text
    name = result.find("a", class_="link-player")
    info = [rating, name]
    print(info)

The HTML parsed is attached in the picture

CodePudding user response：

I tinkered around a little bit and I think I got a version that does what you want

from bs4 import BeautifulSoup
import requests

page = requests.get("https://www.fifaindex.com/players/? 
gender=0&league=1&order=desc")
soup = BeautifulSoup(page.content, "html.parser")
results = soup.find_all("tr")

for result in results:
    try:
        result["data-playerid"]
    except KeyError:
        continue

    rating = result.find("span", class_="badge badge-dark rating r3").text
    name = result.find("a", class_="link-player")
    info = [rating, name]
    print(info)

CodePudding user response：

Get all table lines with a data-playerid attribute will fix it:

#!/usr/bin/env python3
from bs4 import BeautifulSoup
import requests
url = "https://www.fifaindex.com/players/?gender=0&league=1&order=desc"
r = requests.get(url)

soup = BeautifulSoup(r.content, 'html.parser')

results = soup.find_all('tr', {'data-playerid': True})

for res in results:
    rating = res.find("span", class_="badge badge-dark rating r3").text
    name = res.find("a", class_="link-player")
    info = [rating, name]
    print(info)