I am trying to scrape the table from: https://worldpopulationreview.com/states

My code:

from bs4 import BeautifulSoup
import requests
import pandas as pd
url = 'https://worldpopulationreview.com/states'
page = requests.get(url)
soup = BeautifulSoup(page.text,'lxml')
table = soup.find('table', {'class': 'jsx-a3119e4553b2cac7 table is-striped is-hoverable is-fullwidth tp-table-body is-narrow'})
headers = []

for i in table.find_all('th'):
    title = i.text.strip()
    headers.append(title)

df = pd.DataFrame(columns=headers)

for row in table.find_all('tr')[1:]:
    data = row.find_all('td')
    row_data = [td.text.strip() for td in data]
    length = len(df)
    df.loc[length] = row_data

df

Currently returns

'NoneType' object has no attribute 'find_all'

Clearly the error is because the table variable is returning nothing, but I believe I have the table tag correct.

CodePudding user response：

The table data is dynamically loaded by JavaScript and bs4 can't render JS but you can do the job bs4 with an automation tool something like selenium and grab the table using pandas DataFrame.

from selenium import webdriver
import time
from bs4 import BeautifulSoup
import pandas as pd
from selenium.webdriver.chrome.service import Service

webdriver_service = Service("./chromedriver") #Your chromedriver path
driver = webdriver.Chrome(service=webdriver_service)

driver.get('https://worldpopulationreview.com/states')
driver.maximize_window()
time.sleep(8)


soup = BeautifulSoup(driver.page_source,"lxml")


#You can pull the table directly from the web page
df = pd.read_html(str(soup))[0]
print(df)

#OR
#table= soup.select_one('table[]')
# df = pd.read_html(str(table))[0]
# print(df)

Output:

     Rank           State  2022 Population Growth Rate  ...  2010 Population  Growth Since 2010 % of US Density (/mi²)
0      1      California         39995077       0.57%  ...         37253956              7.36%  11.93%            257
1      2           Texas         29945493       1.35%  ...         25145561             19.09%   8.93%            115
2      3         Florida         22085563       1.25%  ...         18801310             17.47%   6.59%            412
3      4        New York         20365879       0.41%  ...         19378102              5.10%   6.07%            432
4      5    Pennsylvania         13062764       0.23%  ...         12702379              2.84%   3.90%            292
5      6        Illinois         12808884      -0.01%  ...         12830632             -0.17%   3.82%            231
6      7            Ohio         11852036       0.22%  ...         11536504              2.74%   3.53%            290
7      8         Georgia         10916760       0.95%  ...          9687653             12.69%   3.26%            190
8      9  North Carolina         10620168       0.86%  ...          9535483             11.38%   3.17%            218
9     10        Michigan         10116069       0.19%  ...          9883640              2.35%   3.02%            179
10    11      New Jersey          9388414       0.53%  ...          8791894              6.78%   2.80%           1277
11    12        Virginia          8757467       0.73%  ...          8001024              9.45%   2.61%            222
12    13      Washington          7901429       1.26%  ...          6724540             17.50%   2.36%            119
13    14         Arizona          7303398       1.05%  ...          6392017             14.26%   2.18%             64
14    15   Massachusetts          7126375       0.68%  ...          6547629              8.84%   2.13%            914
15    16       Tennessee          7023788       0.81%  ...          6346105             10.68%   2.09%            170
16    17         Indiana          6845874       0.44%  ...          6483802              5.58%   2.04%            191
17    18        Maryland          6257958       0.65%  ...          5773552              8.39%   1.87%            645
18    19        Missouri          6188111       0.27%  ...          5988927              3.33%   1.85%             90
19    20       Wisconsin          5935064       0.35%  ...          5686986              4.36%   1.77%            110
20    21        Colorado          5922618       1.27%  ...          5029196             17.76%   1.77%             57
21    22       Minnesota          5787008       0.70%  ...          5303925              9.11%   1.73%             73
22    23  South Carolina          5217037       0.95%  ...          4625364             12.79%   1.56%            174
23    24         Alabama          5073187       0.48%  ...          4779736              6.14%   1.51%            100
24    25       Louisiana          4682633       0.27%  ...          4533372              3.29%   1.40%            108
25    26        Kentucky          4539130       0.37%  ...          4339367              4.60%   1.35%            115
26    27          Oregon          4318492       0.95%  ...          3831074             12.72%   1.29%             45
27    28        Oklahoma          4000953       0.52%  ...          3751351              6.65%   1.19%             58
28    29     Connecticut          3612314       0.09%  ...          3574097              1.07%   1.08%            746
29    30            Utah          3373162       1.53%  ...          2763885             22.04%   1.01%             41
30    31            Iowa          3219171       0.45%  ...          3046355              5.67%   0.96%             58
31    32          Nevada          3185426       1.28%  ...          2700551             17.95%   0.95%             29
32    33        Arkansas          3030646       0.32%  ...          2915918              3.93%   0.90%             58
33    34     Mississippi          2960075      -0.02%  ...          2967297             -0.24%   0.88%             63
34    35          Kansas          2954832       0.29%  ...          2853118              3.57%   0.88%             36
35    36      New Mexico          2129190       0.27%  ...          2059179              3.40%   0.64%             18
36    37        Nebraska          1988536       0.68%  ...          1826341              8.88%   0.59%             26
37    38           Idaho          1893410       1.45%  ...          1567582             20.79%   0.56%             23
38    39   West Virginia          1781860      -0.33%  ...          1852994             -3.84%   0.53%             74
39    40          Hawaii          1474265       0.65%  ...          1360301              8.38%   0.44%            230
40    41   New Hampshire          1389741       0.44%  ...          1316470              5.57%   0.41%            155
41    42           Maine          1369159       0.25%  ...          1328361              3.07%   0.41%             44
42    43    Rhode Island          1106341       0.41%  ...          1052567              5.11%   0.33%           1070
43    44         Montana          1103187       0.87%  ...           989415             11.50%   0.33%
8
44    45        Delaware          1008350       0.92%  ...           897934             12.30%   0.30%            517
45    46    South Dakota           901165       0.81%  ...           814180             10.68%   0.27%             12
46    47    North Dakota           800394       1.35%  ...           672591             19.00%   0.24%             12
47    48          Alaska           738023       0.31%  ...           710231              3.91%   0.22%
1
48    49         Vermont           646545       0.27%  ...           625741              3.32%   0.19%             70
49    50         Wyoming           579495       0.23%  ...           563626              2.82%   0.17%
6

[50 rows x 9 columns]

CodePudding user response：

Table is rendered dynamically from JSON that is placed at the end of the source code, so it do not need selenium simply extract the tag and load the JSON - This also includes all additional information from the page:

soup = BeautifulSoup(requests.get('https://worldpopulationreview.com/states').text)

json.loads(soup.select_one('#__NEXT_DATA__').text)['props']['pageProps']['data']

Example

import requests, json
import pandas as pd
from bs4 import BeautifulSoup

soup = BeautifulSoup(requests.get('https://worldpopulationreview.com/states').text)

pd.DataFrame(
    json.loads(soup.select_one('#__NEXT_DATA__').text)['props']['pageProps']['data']
)

Example

Cause there are also additional information, that is used for the map, simply choose columns you need by header.