I am new to python and am trying to parse a table from the given website into a PANDAS DATAFRAME.

I am using modules requests-html, requests, and beautifulSoup.

Here is the website, I would like to gather the table from:

CodePudding user response：

The data you see on the page is embedded inside <script> in form of JavaScript. You can use selenium or parse the data manually from the page. I'm using js2py module to decode the data:

import re
import js2py
import requests
import pandas as pd


url = "https://www.aamc.org/data-reports/workforce/interactive-data/active-physicians-largest-specialties-2019"
html_doc = requests.get(url).text

data = re.search(r"(?s)\$scope\.schools = (.*?);", html_doc).group(1)
data = [{k: v.strip() for k, v in d.items()} for d in js2py.eval_js(data)]

columns = {
    "specialty": "Specialty",
    "one": "Total Active Physicians",
    "two": "Patient Care",
    "three": "Teaching",
    "four": "Research",
    "five": "Other",
}

df = pd.DataFrame(data).rename(columns=columns)
print(df[list(columns.values())].to_markdown(index=False))

Prints: