How to select a div within a div class with beautiful soup?-CodePudding

I'm really new to this and am confused on how I can select information with a particular div class image of html but it is <div data-start="1.."). I don't understand what data-start is considered within the and how I can get the pricing information from that table, it just keeps showing me none as an output. Any help would be much appreciated.

here is my code:

from bs4 import BeautifulSoup
import bs4
import requests
import json
import numpy as np


urls = ['https://filterbuy.com/brand/trion-air-bear-air-filters/20x20x5-air-bear-20x20/?selected_merv=8',
        'https://filterbuy.com/brand/trion-air-bear-air-filters/20x20x5-air-bear-20x20/?selected_merv=11',
        'https://filterbuy.com/brand/trion-air-bear-air-filters/20x20x5-air-bear-20x20/?selected_merv=13']

#scrape elements
for url in urls:
    response = requests.get(url)
    soup = BeautifulSoup(response.content, "html.parser")

#returned values
    product = soup.find("h1", class_="text-center")
    merv = url.rsplit('?', 0)
    price_table= soup.find("??????")
    json_schema = soup.find_all('script', attrs={'type': 'application/ld json'})[1]
    json_file = json.loads(json_schema.get_text())


    for product, merv, in zip(product, merv):
        print(product.getText(), merv, price_table, json_file)
        np.savetxt('products.csv', [p for p in zip(product, json_schema)], delimiter=',', fmt='%s')

CodePudding user response：

The problem is that this table is not in the source code, it is formed from static data using a JS. All three tables exist immediately on the page. You can access tax values in the cart class

url = 'https://filterbuy.com/brand/trion-air-bear-air-filters/20x20x5-air-bear-20x20/?selected_merv=8'
response = requests.get(url)
mervs = BeautifulSoup(response.text, 'lxml').find_all('strong')
for i, cart in enumerate(BeautifulSoup(response.text, 'lxml').find_all('form', class_='cart')):
    for tax in cart.attrs:
        if 'data-price' in tax:
            print(mervs[i].get_text(), cart[tax])

OUTPUT:

MERV 8 47.58
MERV 8 29.73
MERV 8 28.11
MERV 8 27.56
MERV 8 27.03
MERV 11 57.09
MERV 11 29.13
MERV 11 30.92
MERV 11 30.32
MERV 11 30.02
MERV 11 29.73
MERV 13 62.99
MERV 13 34.11
MERV 13 32.54
MERV 13 32.01
MERV 13 31.49