I am trying to learning to web scrape. When I do ctrl U to see the source code I don't see any of the data I want to however if I inspect the page i can find the information in a different body.
I have written the following code:
url = "https://hereford.ttleagues.com/rankings/24018e59-6861-4082-bb62-5455224a3ad8"
result = requests.get(url)
doc = BeautifulSoup(result.text,"html.parser")
table1 = doc.find("body")
I find a body but not the one I am looking for. The body I want to find
Is anybody able to tell me how to find the first body the has the information i want?
CodePudding user response：
The data that's in the body you want comes from an API and is dynamically rendered.
You can query the API endpoint and rebuild the table.
For example:
import datetime
import pandas as pd
import requests
from tabulate import tabulate
headers = {
"Accept": "application/json",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
"Host": "ttleagues-api.azurewebsites.net",
"Referer": "https://hereford.ttleagues.com/",
"Tenant": "hereford.ttleagues.com",
}
player_id = "24018e59-6861-4082-bb62-5455224a3ad8"
url = f"https://ttleagues-api.azurewebsites.net/api/rankings/{player_id}/stats"
results = requests.get(url, headers=headers).json()["results"]
stats = []
for result in results:
stats.append(
[
result["opponent"]["value"],
result["versus"]["value"],
datetime.datetime
.strptime(result["date"], "%Y-%m-%dT%H:%M:%SZ")
.strftime("%d %b %Y"),
result["for"],
result["against"],
" ".join(f"{g['for']}:{g['against']}" for g in result["games"]),
]
)
df = pd.DataFrame(
stats,
columns=["Opponent", "Versus", "Date", "For", "Against", "Game Scores"],
)
print(tabulate(df, headers="keys", tablefmt="psql", showindex=False))
Output:
------------------------ --------------- ------------- ------- ----------- -----------------------------
| Opponent | Versus | Date | For | Against | Game Scores |
|------------------------ --------------- ------------- ------- ----------- -----------------------------|
| Trevor Davies | Beekeepers | 08 Nov 2022 | 3 | 0 | 15:13 11:1 11:8 |
| Lincoln Franklin | Beekeepers | 08 Nov 2022 | 3 | 2 | 8:11 12:10 11:6 9:11 11:9 |
| Stephen Brown | Beekeepers | 08 Nov 2022 | 2 | 3 | 3:11 11:4 9:11 12:10 14:16 |
| Andrew Reece | Panthers | 25 Oct 2022 | 3 | 0 | 11:1 11:6 11:7 |
| Mark Playle | Panthers | 25 Oct 2022 | 3 | 0 | 11:6 11:8 11:5 |
| Harmesh Jutle | Panthers | 25 Oct 2022 | 3 | 1 | 11:4 8:11 11:2 11:9 |
| Paul Oatley | Warriors | 19 Oct 2022 | 3 | 1 | 11:5 11:8 7:11 11:8 |
| Toby Eckley | Warriors | 19 Oct 2022 | 0 | 3 | 9:11 8:11 7:11 |
| Ola Andersson | Warriors | 19 Oct 2022 | 1 | 3 | 5:11 3:11 12:10 7:11 |
| Andy Castle | Dodgers | 11 Oct 2022 | 3 | 0 | 11:8 11:7 11:6 |
| Alan Smith | Dodgers | 11 Oct 2022 | 3 | 0 | 11:8 11:4 11:8 |
| Sam Turner | Dodgers | 11 Oct 2022 | 3 | 2 | 12:14 11:9 14:12 13:15 11:8 |
| Patryk Kowacz | Pioneers | 06 Oct 2022 | 0 | 3 | 10:12 7:11 7:11 |
| Julian Clapp | Pioneers | 06 Oct 2022 | 3 | 0 | 11:5 11:5 11:4 |
| Piotr Sniezek | Pioneers | 06 Oct 2022 | 0 | 3 | 10:12 10:12 7:11 |
| Zak Weaver | Outcasts | 27 Sep 2022 | 3 | 2 | 7:11 11:9 9:11 11:6 11:7 |
| Dylan Day | Outcasts | 27 Sep 2022 | 2 | 3 | 11:5 10:12 10:12 11:4 10:12 |
| Simon Morgan | Outcasts | 27 Sep 2022 | 2 | 3 | 11:9 9:11 3:11 11:9 10:12 |
| Stephen Brown | Beekeepers | 22 Sep 2022 | 2 | 3 | 11:9 8:11 12:10 8:11 8:11 |
| Trevor Davies | Beekeepers | 22 Sep 2022 | 1 | 3 | 8:11 9:11 11:6 6:11 |
| Lincoln Franklin | Beekeepers | 22 Sep 2022 | 1 | 3 | 13:11 9:11 7:11 12:14 |
| Huan Davies | Commitments | 03 Mar 2022 | 3 | 0 | 11:4 11:7 11:5 |
| Brian Caldicutt | Commitments | 03 Mar 2022 | 3 | 0 | 11:4 11:7 11:5 |
| Andrew Reece | Commitments | 03 Mar 2022 | 3 | 0 | 11:4 12:10 11:5 |
| Patryk Kowacz | Welly Wangers | 22 Feb 2022 | 2 | 3 | 8:11 13:15 11:6 11:9 5:11 |
| Nigel Foreshew | Welly Wangers | 22 Feb 2022 | 1 | 3 | 5:11 11:5 7:11 7:11 |
| Ken Read | Welly Wangers | 22 Feb 2022 | 0 | 3 | 9:11 8:11 8:11 |
| Radoslaw Mikolajczyk | Pioneers | 15 Feb 2022 | 1 | 3 | 11:7 7:11 8:11 5:11 |
| Yan Liu | Pioneers | 15 Feb 2022 | 3 | 0 | 11:3 11:3 11:9 |
| Harmesh Jutle | Pioneers | 15 Feb 2022 | 3 | 0 | 11:6 11:4 11:6 |
| Rod Davies | Magpies | 02 Feb 2022 | 1 | 3 | 11:7 9:11 7:11 7:11 |
| Toby Eckley | Magpies | 02 Feb 2022 | 2 | 3 | 12:10 4:11 11:3 6:11 7:11 |
| Paul Oatley | Magpies | 02 Feb 2022 | 3 | 0 | 11:5 11:7 11:8 |
| Waldemar Kogut | Panthers | 25 Jan 2022 | 1 | 3 | 11:9 8:11 9:11 8:11 |
| Pratikcha Chamling Rai | Panthers | 25 Jan 2022 | 3 | 0 | 11:1 13:11 11:6 |
| Harmesh Jutle | Panthers | 25 Jan 2022 | 3 | 0 | 11:7 11:7 11:8 |
| Ian Godding | Badgers | 19 Jan 2022 | 3 | 0 | 11:4 11:8 11:9 |
| Mark Conod | Badgers | 19 Jan 2022 | 3 | 1 | 11:2 13:11 8:11 12:10 |
| Jake Conod | Badgers | 19 Jan 2022 | 3 | 1 | 11:8 7:11 11:7 12:10 |
| Alan Smith | Dodgers | 11 Jan 2022 | 3 | 0 | 11:2 11:9 12:10 |
| Andy Castle | Dodgers | 11 Jan 2022 | 1 | 3 | 12:10 8:11 5:11 5:11 |
| Bryan Farr | Dodgers | 11 Jan 2022 | 3 | 0 | 11:9 11:8 11:6 |
| Dylan Day | Outcasts | 13 Dec 2021 | 1 | 3 | 11:9 3:11 7:11 13:15 |
| Zak Weaver | Outcasts | 13 Dec 2021 | 0 | 3 | 2:11 7:11 10:12 |
| Simon Morgan | Outcasts | 13 Dec 2021 | 2 | 3 | 8:11 5:11 11:6 14:12 1:11 |
| Bob Marshall | Commitments | 30 Nov 2021 | 3 | 0 | 11:8 11:7 11:4 |
| Pratikcha Chamling Rai | Commitments | 30 Nov 2021 | 3 | 0 | 11:2 11:5 11:8 |
| Brian Caldicutt | Commitments | 30 Nov 2021 | 3 | 0 | 11:4 11:7 11:4 |
| Toby Eckley | Magpies | 02 Nov 2021 | 2 | 3 | 11:6 11:13 14:12 15:17 8:11 |
| Chris Martin | Magpies | 02 Nov 2021 | 1 | 3 | 7:11 11:8 6:11 8:11 |
------------------------ --------------- ------------- ------- ----------- -----------------------------