I am trying to scrape both the 'settle' columns in conjunction with the base month and what respective table they are from (from this url: https://www.asxenergy.com.au/futures_nz/A)
I am able to run an html parser, but as soon as I attempt to run something similar to this:
table1 = soup.find(‘table’)
table1
it just comes back with nothing being there. I assume I'm making an error regarding the table tag. Would really appreciate some help!
Ideally I would like to be able to get the data from this table and then store it in a dataframe.
CodePudding user response：
To read tables to panda's DataFrames you can use next example (as @TimRobers said, the data is loaded with JavaScript from different URL):
import requests
import pandas as pd
from bs4 import BeautifulSoup
url = "https://www.asxenergy.com.au/futures_nz/dataset"
soup = BeautifulSoup(requests.get(url).content, "html.parser")
for table in soup.select("table:not(:has(table))"):
df = pd.read_html(str(table))[0]
df["TITLE"] = table.find_previous("h2").get_text(strip=True)
print(df)
print("-" * 160)
Prints:
Base Month Bid Size Bid Ask Ask Size High Low Last /- Vol OpenInt OpenInt /- Settle TITLE
0 Aug 2022 - - - - 54.00 52.85 54.00 1.45 30 1610 - 52.55 Otahuhu
1 Sep 2022 - - - - 69.00 66.00 66.00 1.00 97 1624 - 65.00 Otahuhu
2 Oct 2022 - - - - 84.10 81.75 81.75 0.30 62 1585 - 81.45 Otahuhu
3 Nov 2022 - - - - 104.00 100.45 100.45 0.40 62 1192 - 100.05 Otahuhu
4 Dec 2022 - - - - 87.25 84.70 84.70 0.35 32 952 - 84.35 Otahuhu
5 Jan 2023 - - - - 119.10 118.10 118.20 0.55 58 524 - 117.65 Otahuhu
6 Feb 2023 - - - - - - - - - 3 - 175.25 Otahuhu
7 Mar 2023 - - - - - - - - - - - 184.20 Otahuhu
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Base Quarter Bid Size Bid Ask Ask Size High Low Last /- Vol OpenInt OpenInt /- Settle TITLE
0 Q3 2022 - - - - 75.30 73.15 73.65 -0.65 31 3679 - 74.30 Otahuhu
1 Q4 2022 - - - - 91.60 87.60 88.50 - 65 4109 - 88.50 Otahuhu
2 Q1 2023 - - - - 163.25 158.50 158.50 - 123 3401 - 158.50 Otahuhu
3 Q2 2023 - - - - - - - - - 2403 - 214.00 Otahuhu
4 Q3 2023 - - - - 216.00 216.00 216.00 - 30 2438 - 216.00 Otahuhu
5 Q4 2023 - - - - 143.55 142.00 143.55 - 60 3357 - 143.55 Otahuhu
6 Q1 2024 - - - - - - - - - 3093 - 159.00 Otahuhu
7 Q2 2024 - - - - 197.00 197.00 197.00 - 30 2082 - 197.00 Otahuhu
8 Q3 2024 - - - - - - - - - 2091 - 197.50 Otahuhu
9 Q4 2024 - - - - 145.95 143.00 145.95 - 40 2649 - 145.95 Otahuhu
10 Q1 2025 - - - - 151.00 150.50 150.70 -0.30 46 1838 - 151.00 Otahuhu
11 Q2 2025 - - - - 178.00 175.20 176.00 - 92 1619 - 176.00 Otahuhu
12 Q3 2025 - - - - 178.00 175.20 176.00 - 92 1316 - 176.00 Otahuhu
13 Q4 2025 - - - - 128.45 125.00 125.50 - 100 1845 - 125.50 Otahuhu
----------------------------------------------------------------------------------------------------------------------------------------------------------------
...and so on.