how can i get all product name by scraping this-CodePudding

import requests as r
from bs4 import BeautifulSoup as bs
url=r.get("https://www.consumerreports.org/cro/coffee-makers.htm")
soup=bs(url.content)
product=soup.find('div',class_="row product-type-container")
pclass=product.find('div',class_="product-type-item col-xs-4")
pname=pclass.find('div',class_="product-type-info-container").h3.text
print(pname)

i am scraping all the product name and details but can only scrape one product at a time how can i scrape

CodePudding user response：

To get titles of all products in all categories you can use next example:

import requests
from bs4 import BeautifulSoup


def get_products(url):
    soup = BeautifulSoup(requests.get(url).content, "html.parser")

    out = []
    for title in soup.select(".crux-component-title"):
        out.append(title.get_text(strip=True))

    return out


url = "https://www.consumerreports.org/cro/coffee-makers.htm"
soup = BeautifulSoup(requests.get(url).content, "html.parser")

all_data = []
for category_link in soup.select("h3.crux-product-title a"):
    u = "https://www.consumerreports.org"   category_link["href"]
    print("Getting {}".format(u))
    all_data.extend(get_products(u))

for i, title in enumerate(all_data, 1):
    print("{:<5} {}".format(i, title))

Prints:

1     Bella 14755 with Brew Strength Selector
2     Bella Pro Series 90061
3     Betty Crocker 12-cup Stainless Steel BC-2809CB
4     Black Decker 12-cup Programmable CM1331S
5     Black Decker 12-Cup Thermal Programmable CM2046S
6     Black Decker CM2036S 12-cup Thermal
7     Black Decker CM4000S
8     Black Decker DLX1050B
9     Black Decker Even Stream CM2035B
10    Black Decker Honeycomb Collection CM1251W
11    Black Decker Programmable CM1331BS (Walmart Exclusive)
12    Bonavita BV1901TS 8-Cup One-Touch
13    Braun Brew Sense KF7150BK
14    Braun BrewSense 12-cup Programmable KF7150
15    Braun BrewSense 12-cup Programmable KF7000BK

...and so on.

CodePudding user response：

Why is that: find(..) returns only first object which matches your criteria.

Solution: Try using find_all(..) method.