Can't scrape two fields simultaneously from a webpage-CodePudding

I'm trying to scrape series name along with product code from a webpage. The script that I have created can fetch the product code flawlessly but I can't find any idea how to fetch the series name along with the product code.

I've tried so far:

import requests
from bs4 import BeautifulSoup

link = 'https://www.theimagingsource.com/products/industrial-cameras/usb-3.1-monochrome/'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36'
    res = s.get(link)
    soup = BeautifulSoup(res.text,"lxml")
    for item in soup.select("td.product-code > a[title]"):
        print(item.get_text(strip=True))

Output I'm getting:

DMK 38UX267
DMK 38UX255
DMK 38UX304
DMK 38UX253
DMK 37AUX287
DMK 37AUX273
DMK 37AUX290

Output I wish to get:

38 series - USB 3.1 monochrome industrial cameras DMK 38UX267
38 series - USB 3.1 monochrome industrial cameras DMK 38UX255
38 series - USB 3.1 monochrome industrial cameras DMK 38UX304
38 series - USB 3.1 monochrome industrial cameras DMK 38UX253
37 A series - USB 3.1 monochrome industrial cameras DMK 37AUX287
37 A series - USB 3.1 monochrome industrial cameras DMK 37AUX273
37 A series - USB 3.1 monochrome industrial cameras DMK 37AUX290

and so on ---

CodePudding user response：

Use .find_previous('h3'):

import requests
from bs4 import BeautifulSoup

link = "https://www.theimagingsource.com/products/industrial-cameras/usb-3.1-monochrome/"

with requests.Session() as s:
    s.headers[
        "User-Agent"
    ] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36"
    res = s.get(link)
    soup = BeautifulSoup(res.text, "lxml")
    for item in soup.select("td.product-code > a[title]"):
        print(
            item.find_previous("h3").get_text(strip=True),
            item.get_text(strip=True),
        )

Prints:

38 series - USB 3.1 monochrome industrial cameras DMK 38UX267
38 series - USB 3.1 monochrome industrial cameras DMK 38UX255
38 series - USB 3.1 monochrome industrial cameras DMK 38UX304
38 series - USB 3.1 monochrome industrial cameras DMK 38UX253
37 A series - USB 3.1 monochrome industrial cameras DMK 37AUX287
37 A series - USB 3.1 monochrome industrial cameras DMK 37AUX273
37 A series - USB 3.1 monochrome industrial cameras DMK 37AUX290
37 A series - USB 3.1 monochrome industrial cameras DMK 37AUX252
37 A series - USB 3.1 monochrome industrial cameras DMK 37AUX265
37 A series - USB 3.1 monochrome industrial cameras DMK 37AUX250
37 A series - USB 3.1 monochrome industrial cameras DMK 37AUX264
37 A series - USB 3.1 monochrome industrial cameras DMK 37AUX178
37 A series - USB 3.1 monochrome industrial cameras DMK 37AUX226
37 B series - USB 3.1 monochrome industrial cameras DMK 37BUX287
37 B series - USB 3.1 monochrome industrial cameras DMK 37BUX273
37 B series - USB 3.1 monochrome industrial cameras DMK 37BUX290
37 B series - USB 3.1 monochrome industrial cameras DMK 37BUX252
37 B series - USB 3.1 monochrome industrial cameras DMK 37BUX265
37 B series - USB 3.1 monochrome industrial cameras DMK 37BUX250
37 B series - USB 3.1 monochrome industrial cameras DMK 37BUX264
37 B series - USB 3.1 monochrome industrial cameras DMK 37BUX178
37 B series - USB 3.1 monochrome industrial cameras DMK 37BUX226