Home > OS >  How to use BeautifulSoup for td tags without attributes?
How to use BeautifulSoup for td tags without attributes?

Time:09-01

I am trying to extract this immigration data from Canada immigration website

I tried extracting the whole table, individual td tags, all returns empty list.

from bs4 import BeautifulSoup
from requests_html import HTMLSession
import requests

s = HTMLSession()
data=[]
url = "https://www.canada.ca/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds.html"


def getdata(url):
    r = s.get(url)
    soup = BeautifulSoup(r.text, 'html.parser')
    return soup

def gettabledata(soup):
    table = soup.find_all('td',{'class':'sorting_1'})

I also tried finding tr tags in a table;

table = soup.find('table', class_='table')
print("\n TABLE \n")
print(table)
table_body = table.find('tbody')
print("\n TABLEBODY \n")
print(table_body)

rows = table_body.find_all('tr')
print("\n ROWS \n")
print(rows)
for row in rows:
    cols = row.find_all('td')
    cols = [ele.text.strip() for ele in cols]
    data.append([ele for ele in cols if ele])
print(data)

Also tried , but doesnt work:

divs = soup.find_all('div', class_='dataTables_wrapper')
print("divs \n", divs)
for div in divs:
    table = div.find('tbody',class_='wb-data-json-inited')
    print("\n TABLE \n")
    print(table)
    table_body = table.find('tbody')
    print("\n TABLEBODY \n")
    print(table_body)

    rows = table.find_all('tr')
    print("\n ROWS \n")
    print(rows)
    for row in rows:
        cols = row.find_all('td')
        cols = [ele.text.strip() for ele in cols]
        data.append([ele for ele in cols if ele])
    print(data)

What am I missing, and how can I extract the table data?

Another doubt I have is how I can access the Program name and values just enclosed in td tags with no attributes. Each line in the table is a tr tag with a bunch of td tags in them :

        <tr>
            <td><a href="/content/canadasite/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-219.html">219</a></td>
            <td >March 30, 2022</td>
            <td>Provincial Nominee Program</td>
            <td>919</td>
            <td>785</td>
            <td>30-03-2022</td>
            <td><p>Provincial Nominee Program</p></td>
        </tr>

CodePudding user response:

That table is being loaded dynamically in page. You can inspect the Network Tab in Dev Tools, and see which API is being accessed to pull the data from. This is one way of obtaining the data in that table:

import requests
import pandas as pd

r = requests.get('https://www.canada.ca/content/dam/ircc/documents/json/ee_rounds_123_en.json')
df = pd.DataFrame(r.json()['rounds'])
print(df)

This will return a dataframe:

    drawNumber  drawNumberURL   drawDate    drawDateFull    drawName    drawSize    drawCRS mitext  DrawText1   drawText2   drawDateTime    drawCutOff  drawDistributionAsOn    dd1 dd2 dd3 dd4 dd5 dd6 dd7 dd8 dd9 dd10    dd11    dd12    dd13    dd14    dd15    dd16    dd17    dd18
0   229 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=229'>229</a>  2022-08-17  August 17, 2022 No Program Specified    2,250   525 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=229'>Invitations to apply for permanent residence under the Express Entry system #229</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=229'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program    August 17, 2022 at 13:43:47 UTC December 28, 2021 at 11:03:15 UTC   August 15, 2022 538 8,221   62,753  5,435   9,129   18,831  16,465  12,893  58,113  12,200  12,721  9,801   11,138  12,253  68,440  35,745  5,137   238,947
1   228 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=228'>228</a>  2022-08-03  August 3, 2022  No Program Specified    2,000   533 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=228'>Invitations to apply for permanent residence under the Express Entry system #228</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=228'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program    August 03, 2022 at 15:16:24 UTC January 06, 2022 at 14:29:50 UTC    August 2, 2022  640 8,975   62,330  5,343   9,044   18,747  16,413  12,783  57,987  12,101  12,705  9,747   11,117  12,317  68,325  35,522  5,145   238,924
2   227 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=227'>227</a>  2022-07-20  July 20, 2022   No Program Specified    1,750   542 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=227'>Invitations to apply for permanent residence under the Express Entry system #227</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=227'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program    July 20, 2022 at 16:32:49 UTC   December 30, 2021 at 15:29:35 UTC   July 18, 2022   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
3   226 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=226'>226</a>  2022-07-06  July 6, 2022    No Program Specified    1,500   557 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=226'>Invitations to apply for permanent residence under the Express Entry system #226</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=226'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program    July 6, 2022 at 14:34:34 UTC    November 13, 2021 at 02:20:46 UTC   July 11, 2022   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
4   225 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=225'>225</a>  2022-06-22  June 22, 2022   Provincial Nominee Program  636 752 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=225'>Invitations to apply for permanent residence under the Express Entry system #225</a> <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations.html?q=225'>See full text of Ministerial Instruction</a> Provincial Nominee Program  June 22, 2022 at 14:13:57 UTC   April 19, 2022 at 13:45:45 UTC  June 20, 2022   664 8,017   55,917  4,246   7,845   16,969  15,123  11,734  53,094  10,951  11,621  8,800   10,325  11,397  64,478  33,585  4,919   220,674
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
225 5   <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-5.html'>5</a>    2015-03-20  March 20, 2015  No Program Specified    1,620   481 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-5.html'>Invitations to apply for permanent residence under the Express Entry system #5</a>   <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-5.html'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program    March 20, 2015 at 23:59:53 UTC      March 20, 2015  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
226 4   <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-4.html'>4</a>    2015-02-27  February 27, 2015   No Program Specified    1,187   735 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-4.html'>Invitations to apply for permanent residence under the Express Entry system #4</a>   <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-4.html'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program    February 27, 2015 at 23:59:51 UTC       February 27, 2015   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
227 3   <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-3.html'>3</a>    2015-02-20  February 20, 2015   Canadian Experience Class   849 808 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-3.html'>Invitations to apply for permanent residence under the Express Entry system #3</a>   <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-3.html'>See full text of Ministerial Instruction</a> Canadian Experience Class   February 20, 2015 at 11:59:47 UTC       February 20, 2015   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
228 2   <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-2.html'>2</a>    2015-02-07  February 7, 2015    No Program Specified    779 818 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-2.html'>Invitations to apply for permanent residence under the Express Entry system #2</a>   <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-2.html'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program    February 7, 2015 at 11:59:59 UTC        February 7, 2015    0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
229 1   <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-1.html'>1</a>    2015-01-31  January 31, 2015    No Program Specified    779 886 <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-1.html'>Invitations to apply for permanent residence under the Express Entry system #1</a>   <a href='/en/immigration-refugees-citizenship/corporate/mandate/policies-operational-instructions-agreements/ministerial-instructions/express-entry-rounds/invitations-1.html'>See full text of Ministerial Instruction</a> Federal Skilled Worker, Canadian Experience Class, Federal Skilled Trades and Provincial Nominee Program    January 31, 2015 at 11:59:48 UTC        January 31, 2015    0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0
230 rows × 31 columns
  • Related