I'm using BeautifulSoup and trying to scrap a HTML table. I'm only interested in the first table. However, the output is missing one column values - the "Entries" column. Not sure what I'm doing wrong here.

Here's my code:

import requests
from bs4 import BeautifulSoup

URL = "http://www.godaycare.com/child-care-cost/saskatchewan"
page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")

table = soup.find_all('table')[0]

for child in soup.find_all('table')[0].children:
    for td in child:
        print(td.text)

Here's the output

TypeAge Cat.SpotAVG. Cost ($)Entries
LicensedInfantFull-Time751.02717
LicensedInfantPart-Time41.31187
UnlicensedInfantFull-Time699.56287
UnlicensedInfantPart-Time31.0550
LicensedToddlerFull-Time661.04604
LicensedToddlerPart-Time32.69148
UnlicensedToddlerFull-Time633.01342
UnlicensedToddlerPart-Time35.9969
LicensedPreschoolFull-Time595.45327
LicensedPreschoolPart-Time30.8566
UnlicensedPreschoolFull-Time602.82195
UnlicensedPreschoolPart-Time30.3330
LicensedKindergartenFull-Time562.8787
LicensedKindergartenPart-Time28.2938
UnlicensedKindergartenFull-Time549.1257
UnlicensedKindergartenPart-Time23.0113
LicensedSchoolageFull-Time605.3494
LicensedSchoolagePart-Time25.4533
UnlicensedSchoolageFull-Time434.9098
UnlicensedSchoolagePart-Time19.0025

CodePudding user response：

The easiest way how to read the first table is using pandas.read_html:

import pandas as pd

url = "http://www.godaycare.com/child-care-cost/saskatchewan"

df = pd.read_html(url)[0]
print(df.to_markdown())

Prints:

EDIT: Version with beautifulsoup:

import requests
from bs4 import BeautifulSoup

URL = "http://www.godaycare.com/child-care-cost/saskatchewan"
page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")

for row in soup.find("table").find_all("tr"):
    tds = [td.text for td in row.find_all(["td", "th"])]
    print(("{:<20}" * len(tds)).format(*tds))

Prints:

Type                Age Cat.            Spot                AVG. Cost ($)       Entries             
Licensed            Infant              Full-Time           751.02              717                 
Licensed            Infant              Part-Time           41.31               187                 
Unlicensed          Infant              Full-Time           699.56              287                 
Unlicensed          Infant              Part-Time           31.05               50                  
Licensed            Toddler             Full-Time           661.04              604                 
Licensed            Toddler             Part-Time           32.69               148                 
Unlicensed          Toddler             Full-Time           633.01              342                 
Unlicensed          Toddler             Part-Time           35.99               69                  
Licensed            Preschool           Full-Time           595.45              327                 
Licensed            Preschool           Part-Time           30.85               66                  
Unlicensed          Preschool           Full-Time           602.82              195                 
Unlicensed          Preschool           Part-Time           30.33               30                  
Licensed            Kindergarten        Full-Time           562.87              87                  
Licensed            Kindergarten        Part-Time           28.29               38                  
Unlicensed          Kindergarten        Full-Time           549.12              57                  
Unlicensed          Kindergarten        Part-Time           23.01               13                  
Licensed            Schoolage           Full-Time           605.34              94                  
Licensed            Schoolage           Part-Time           25.45               33                  
Unlicensed          Schoolage           Full-Time           434.90              98                  
Unlicensed          Schoolage           Part-Time           19.00               25