I am using BeautifulSoup to scrape the data from the website link. There are total 9 table with same class name but I am only able to get link 5 tables. What changes should I make in the code so I can scrape all the tables present from the above link?

Below is the code I have used:

def ScrapeSecScreen():
    options = webdriver.ChromeOptions()
    options.add_argument('--ignore-certificate-errors')
    options.add_argument('--incognito')
    options.add_argument('--headless')
    driver = webdriver.Chrome("C:/Users/pralo/Downloads/DE/chromedriver", chrome_options=options)
    driver.get('https://www.panamacompra.gob.pa/Inicio/v2/#!/vistaPreviaCP?NumLc=2022-0-30-0-08-CL-024792&esap=0&nnc=0&it=1')
    sleep(10)
    sourcecode = driver.execute_script("return document.getElementsByTagName('html')[0].innerHTML")
    # print(sourcecode)
    soup = BeautifulSoup(sourcecode,"html.parser")
    print(soup)
    l1 = []
    tablelist1=soup.findAll('table',{'class':'table table-condensed table-bordered last-line-table'})
    for tr in tablelist1:
        td = tr.find_all('tr')
        row = [tr.text for tr in td]
        l1.append(row)
    print(l1)
ScrapeSecScreen()

CodePudding user response：

Upon inspection, it turns out that only first five tables match the classes you mentioned. To get all tables, you can omit the class argument or specify a class that all the tables share.

Check this line of code:

soup.find_all('table', class_='table')