I am using the following code to get a table from this page. However, I want to include an additional column that parses out the URL from the 'href' tag in the final column called Volume Page. This is my first attempt at using python for web scraping, so any assistance would be appreciated.

    import requests
from bs4 import BeautifulSoup as bs
import pandas as pd
url = 'https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/compliance-actions-and-activities/fda-debarment-list-drug-product-applications'
r = requests.get(url)
soup = bs(r.content,'lxml')
table =soup.select('table')[-1]
rows = table.find_all('tr')
output = []
for row in rows:
    cols = row.find_all('td')
    cols = [item.text.strip() for item in cols]
    output.append([item for item in cols if item])

df = pd.DataFrame(output, columns = ['Last Name', 'First Name','Effective Date','End Term','FR Date', 'Volume Page'])
df = df.iloc[1:]
print(a)
print(df)

CodePudding user response：

Another version:

import requests
import pandas as pd
from bs4 import BeautifulSoup


url = "https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/compliance-actions-and-activities/fda-debarment-list-drug-product-applications"

soup = BeautifulSoup(requests.get(url).content, "html.parser")
table = soup.select("table")[-1]
df = pd.read_html(str(table))[0]

df["URL"] = [tr.a["href"] if tr.a else "-" for tr in table.select("tr:has(td)")]

print(df.to_markdown())

Prints:

...

CodePudding user response：

Simply append() the url if available to your cols:

...
for row in rows:
    cols = row.find_all('td')
    cols = [item.text.strip() for item in cols]
    if row.a:
        cols.append(row.a.get('href'))
    else:
        cols.append('None')
    output.append(cols)
...

Output: