I want to fetch all the details of wrestlers from the tables-CodePudding

I have a link its this- www.cagematch.net/?id=8&nr=1&page=15

In this link you will able to see a table with wrestlers. But If you click on the the name of a wrestler you will be able to see details of a wrestler. So, I want to fetch all the wrestlers with details in an easy & shortcut way. In my mind, I am thinking like this :

urls = [
 link1, link2, link3, link4
]

for u in urls:
 ..... do the scrap

But there are 275 wrestlers I don't want to enter all the links like this. Is there any easy way to do it?

CodePudding user response：

To get all links into a list and then info about each wrestler you can use this example:

import requests
from bs4 import BeautifulSoup


url = "http://www.cagematch.net/?id=8&nr=1&page=15"
headers = {"Accept-Encoding": "deflate"}
soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")

links = [
    "https://www.cagematch.net/"   a["href"] for a in soup.select(".TCol a")
]

for u in links:
    soup = BeautifulSoup(
        requests.get(u, headers=headers).content, "html.parser"
    )
    print(soup.h1.text)
    for info in soup.select(".InformationBoxRow"):
        print(
            info.select_one(".InformationBoxTitle").text.strip(),
            info.select_one(".InformationBoxContents").text.strip(),
        )
    # get other info here
    # ...

    print("-" * 80)

Prints:

Adam Pearce
Current gimmick: Adam Pearce
Age: 44 years
Promotion: World Wrestling Entertainment
Active Roles: Road Agent, Trainer, On-Air Official, Backstage Helper
Birthplace: Lake Forest, Illinois, USA
Gender: male
Height: 6' 2" (188 cm)
Weight: 238 lbs (108 kg)
WWW: http://twitter.com/ScrapDaddyAP https://www.facebook.com/OfficialAdamPearce https://www.youtube.com/watch?v=us91bK1ScL4
Alter egos: Adam O'BrienAdam Pearce    a.k.a.  US Marshall Adam J. PearceMasked Spymaster #2Tommy Lee Ridgeway
Roles: Singles Wrestler (1996 - 2014)Road Agent (2015 - today)Booker (2008 - 2010)Trainer (2013 - today)On-Air Official (2020 - today)Backstage Helper (2015 - today)
Beginning of in-ring career: 16.05.1996
End of in-ring career: 21.12.2014
In-ring experience: 18 years
Wrestling style: Allrounder
Trainer: Randy Ricci & Sonny Rogers
Nicknames: "Scrap Iron"
Signature moves: PiledriverFlying Body SplashRackbomb II
--------------------------------------------------------------------------------
AJ Styles
Current gimmick: AJ Styles
Age: 45 years
Promotion: World Wrestling Entertainment
Brand: RAW
Active Roles: Singles Wrestler
Birthplace: Jacksonville, North Carolina, USA
Gender: male
Height: 5' 11" (180 cm)
Weight: 218 lbs (99 kg)
Background in sports: Ringen, Football, Basketball, Baseball
WWW: http://AJStyles.org https://www.facebook.com/AJStylesOrg-110336188978264/ https://twitter.com/AJStylesOrg https://www.instagram.com/ajstylesp1/ https://www.twitch.tv/Stylesclash
Alter egos: AJ Styles    a.k.a.  Air StylesMr. Olympia
Roles: Singles Wrestler (1999 - today)Tag Team Wrestler (2001 - 2021)
Beginning of in-ring career: 15.02.1999
In-ring experience: 23 years
Wrestling style: Techniker, High Flyer
Trainer: Rick Michaels
Nicknames: "The Phenomenal""The Prince Of Phenomenal"
Signature moves: Styles ClashPelé KickCalf Killer/Calf CrusherStylin' DDTCliffhangerSpiral TapPhenomenal Forearm450 Splash
--------------------------------------------------------------------------------

...and so on.