How to get two element from two dropdown lists (webscraping with python)-CodePudding

I want to webscrape this webpage (www.autocar.co.uk). Therefore, I would like to select each Automaker in a dropdown menu and the model. I always want to skip the "All models" option.

As I just started coding I would higly appreciate your input!

Desired output:

Auto OEM (e.g, Tesla)
   All Models of Tesla (e.g. Model 3, Y...) 

Example:
    Abarth
       595
       595 Competi...
       124 Spider...
       695 Bopisto...
    AC Cars
       #Skip due to no model

My code as of now:

from bs4 import BeautifulSoup
import requests


#Inputs/URLs to scrape: 
URL = ('https://www.autocar.co.uk/car-review/tesla')
(response := requests.get(URL)).raise_for_status()
soup = BeautifulSoup(response.text, 'lxml')
overview = soup.find()

oems = soup.select('select.car-finder-make-chooser option')                      
for oem_loop in oems[1:]:
    print(oem_loop.text)
    models = soup.select('select.car-finder-model-chooser option')  
    for model_loop in models:
        print(model_loop)

My output as of now:

Abarth
<option value="0">Model</option>
AC Cars
<option value="0">Model</option>
AC Schnitzer
<option value="0">Model</option>
Aiways
<option value="0">Model</option>
Allard
<option value="0">Model</option>
Alfa Romeo
<option value="0">Model</option>
Alpina
<option value="0">Model</option>
Alpine
<option value="0">Model</option>
Ariel
<option value="0">Model</option>
Ascari
<option value="0">Model</option>
Aston Martin
<option value="0">Model</option>
Audi
<option value="0">Model</option>
BAC
<option value="0">Model</option>
Bentley
<option value="0">Model</option>
Bizzarrini
<option value="0">Model</option>
BMW
<option value="0">Model</option>
Borgward
<option value="0">Model</option>
Bowler
<option value="0">Model</option>
Bugatti
<option value="0">Model</option>
BYD
<option value="0">Model</option>
...

CodePudding user response：

The following code will get you a list of makes, models and their corresponding urls, and it will save a csv file with the details:

import requests
from bs4 import BeautifulSoup 
import pandas as pd

url = "http://www.autocar.co.uk/"
s = requests.Session()

r = s.get(url)
soup = BeautifulSoup(r.text,'html.parser')
full_car_list = []
car_list = [(x.text, x.get("value"), f'https://www.autocar.co.uk/ajax/car-models/{x.get("value")}/0') for x in soup.select_one('#edit-make').select('option')]
for x in car_list:
    r = s.get(x[2])
    try:
        for item in r.json()['options'].items():
            full_car_list.append((x[0], item[1], f'https://www.autocar.co.uk{item[0]}'))
    except Exception as e:
        full_car_list.append((x[0], 'no models', f'https://www.autocar.co.uk/vehicles/{x[0]}'))
cars_df = pd.DataFrame(full_car_list[1:], columns = ['Make', 'Model', 'Url'])
cars_df = cars_df[cars_df.Model != 'All models']
cars_df.to_csv('makes_models.csv')
print(cars_df.head(15))

This will return a csv file, and will print the head of the dataframe:

Make    Model   Url
1   Abarth  595 https://www.autocar.co.uk/car-review/abarth/595
2   Abarth  595 Competizione    https://www.autocar.co.uk/car-review/abarth/595-competizione
3   Abarth  124 Spider 2016-2019    https://www.autocar.co.uk/car-review/abarth/124-spider-2016-2019
4   Abarth  695 Biposto 2015-2016   https://www.autocar.co.uk/car-review/abarth/695-biposto-2015-2016
5   AC Cars no models   https://www.autocar.co.uk/vehicles/AC Cars
7   AC Schnitzer    ACS3 Sport  https://www.autocar.co.uk/car-review/ac-schnitzer/acs3-sport
8   AC Schnitzer    ACS1    https://www.autocar.co.uk/car-review/ac-schnitzer/acs1
9   AC Schnitzer    ACS5 Sport  https://www.autocar.co.uk/car-review/ac-schnitzer/acs5-sport
10  Aiways  no models   https://www.autocar.co.uk/vehicles/Aiways
12  Allard  J2X MkII    https://www.autocar.co.uk/car-review/allard/j2x-mkii
[...]

I suggest going over the documentation for bs4: https://www.crummy.com/software/BeautifulSoup/bs4/doc/

Also for pandas: https://pandas.pydata.org/docs/

And also for python requests: https://requests.readthedocs.io/en/latest/