How to load more element from web pages using BeautifulSoup and/or selenium-CodePudding

I would like to get each link that the each boxes contain, the page is https://www.quattroruote.it/listino/audi

In this webpage there are all the model that this brand is producing, and each model is a boxes that links to another page (the one in which I should work with).

My problem is that the initial page do not load all the boxes the first time, you have to scroll down and press the red button "Carica altri modelli" (which mean "Load other models"). Is there a way to automatically store in one variable all the links that i need? For example, the first links of the first box is "/listino/audi/a1"

Thanks in advance to anyone who try to help me!!

CodePudding user response：

Not sure exactly what links you want, but you can make the requests iterating through the itemStart parameter.

import requests
from bs4 import BeautifulSoup

for i in range(1,100):
    print('\t\tList start %s' %i)
    url = 'https://www.quattroruote.it/listino/ricerca-more-desktop.html'
    payload = {
        'area': 'NEW',
        'itemStart': '%s' %(i*8),
        '_': '1634219611449'}
    
    response = requests.get(url, params=payload)
    
    soup = BeautifulSoup(response.text, 'html.parser')
    links = soup.find_all('a', href=True)
    
    for link in links:
        print(link['href'])