The table on this page needs to be scraped daily. We are trying to keep the scraping as simple (robust) as possible so there are no issues with the code running on our server. Would like to steer clear of Selenium:

import requests
import pandas as pd

page_list = pd.read_html('https://www.ncaa.com/rankings/basketball-women/d1/ncaa-womens-basketball-net-rankings')
page_df = pd.DataFrame(page_list) 
# won't convert to df (ValueError: Must pass 2-d input. shape=(1, 356, 9)

r = requests.get('https://www.ncaa.com/rankings/basketball-women/d1/ncaa-womens-basketball-net-rankings')
# not sure what to do with response

page_list is close but it is a 3-dimensional list. How can we get this into a 2-dimensional list, or into a pandas dataframe?

CodePudding user response：

pd.read_html doesn't return a DataFrame but a list of dataframes. Use page_list[0] to get the first dataframe:

page_df = pd.DataFrame(page_list[0])

From the documentation:

Read HTML tables into a list of DataFrame objects.