So I am scraping a used car website I've got the make, model, year, and miles but I don't know how to get the others due to them being the li tag as well. I've put all my code here

from bs4 import BeautifulSoup
import requests
import pandas as pd
url = 'https://jammer.ie/used-cars'
response = requests.get(url)
response.status_code
soup = BeautifulSoup(response.content, 'html.parser')
soup
results = soup.find_all('div', {'class': 'span-9 right-col'})
len(results)
results[0].find('h6',{'class':'car-make'}).get_text()
results[0].find('p', {'class':'model'}).get_text()
results[0].find('p', {'class': 'year'}).get_text()
results[0].find('li').get_text().replace('\n', "")

I get the information I want from the above code but for other parts of the li tags they have img tags and span tags how can I get the information from each of the li tags? I am new to python so would like it to be somewhat simply and explained to me please

I tired using the img tag but don't think I used it right.

CodePudding user response：

To get all features into a dataframe you can do:

import requests
import pandas as pd
from bs4 import BeautifulSoup


url = "https://jammer.ie/used-cars"
soup = BeautifulSoup(requests.get(url).text, "html.parser")

all_data = []
for car in soup.select(".car"):
    info = car.select_one(".top-info").get_text(strip=True, separator="|")
    make, model, year, price = info.split("|")

    features = {}
    for feature in car.select(".car--features li"):
        k = feature.img["src"].split("/")[-1].split(".")[0]
        v = feature.span.text
        features[f"feature_{k}"] = v

    all_data.append(
        {"make": make, "model": model, "year": year, "price": price, **features}
    )

df = pd.DataFrame(all_data)
print(df.to_markdown(index=False))

Prints: