How to scrape tables using Beautiful Soup?-CodePudding

I tried scraping tables according to the question: Python BeautifulSoup scrape tables

From the top solution, there I tried:

HTML code:

<div class="table-frame small">
    <table id="rfq-display-line-items-list" class="table">
        <thead id="rfq-display-line-items-header">
          <tr>
          <th>Mfr. Part/Item #</th>
          <th>Manufacturer</th>
          <th>Product/Service Name</th>
          <th>Qty.</th>
          <th>Unit</th>
          <th>Ship Address</th>
        </tr>
      </thead>
      <tbody id="rfq-display-line-item-0">

        <tr>
            <td><span class="small">43933</span></td>
            <td><span class="small">Anvil International</span></td>
            <td><span class="small">Cap Steel Black 1-1/2"</span></td>
            <td><span hljs-string">">800</span></td>
            <td><span hljs-string">">EA</span></td>
            <td><span hljs-string">">1</span></td>
        </tr>
      <!----><!---->
      </tbody><tbody id="rfq-display-line-item-1">

        <tr>
            <td><span hljs-string">">330035205</span></td>
            <td><span hljs-string">">Anvil International</span></td>
            <td><span hljs-string">">1-1/2" x 8" Black Steel Nipple</span></td>
            <td><span hljs-string">">400</span></td>
            <td><span hljs-string">">EA</span></td>
            <td><span hljs-string">">1</span></td>
        </tr>
      <!----><!---->
      </tbody><!---->
    </table><!---->
</div>

According to solution ,

What I tried is:

for tr in soup.find_all('table', {'id': 'rfq-display-line-items-list'}):
    tds = tr.find_all('td')
    print(tds[0].text, tds[1].text, tds[2].text, tds[3].text, tds[4].text, tds[5].text)

But this displayed only the first row,

43933 Anvil International Cap Steel Black 1-1/2" 800 EA 1

I later out found out the all those <td> were stored in the list. I want to print all the rows.

Expected Output:

43933      Anvil International Cap Steel Black 1-1/2" 800 EA 1
330035205  Anvil International 1-1/2" x 8" Black Steel Nipple 400 EA 1

CodePudding user response：

You start with tr tag & go down to td

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")

for tr in soup.find("table", id="rfq-display-line-items-list").find_all("tr"):
    print(" ".join([td.text for td in tr.find_all('td')]))

43933 Anvil International Cap Steel Black 1-1/2" 800 EA 1
330035205 Anvil International 1-1/2" x 8" Black Steel Nipple 400 EA 1

CodePudding user response：

You can do that using css selectors as follows:

for tr in soup.select('table#rfq-display-line-items-list tbody tr'):
    tds = tr.find_all('td')
    print(tds[0].text, tds[1].text, tds[2].text, tds[3].text, tds[4].text, tds[5].text)

output:

43933      Anvil International Cap Steel Black 1-1/2" 800 EA 1
330035205  Anvil International 1-1/2" x 8" Black Steel Nipple 400 EA 1