Home > OS >  How to scrape data from multiple pages using beautifulsoup
How to scrape data from multiple pages using beautifulsoup

Time:11-15

I am try to scrape multiple page but they give me nothing kindly help me to resolve these issue

    import requests
    from bs4 import BeautifulSoup
    import pandas as pd
    headers= {'User-Agent': 'Mozilla/5.0'}
    for page in range(1,2 ):
        response = requests.get("https://www.avbuyer.com/aircraft/private-jets={page}".format(
                page=page
            ),
            headers=headers,
        )
    
        soup = BeautifulSoup(response.content, 'html.parser')
        postings = soup.find_all('div', class_ = 'listing-item premium')
        for post in postings:
            link = post.find('a', class_ = 'more-info').get('href')
            link_full = 'https://www.avbuyer.com'  link
            plane = post.find('h2', class_ = 'item-title').text
            price = post.find('div', class_ = 'price').text
            location = post.find('div', class_ = 'list-item-location').text
            print(location)

CodePudding user response:

problem was in page = that should be page- .Now your code is working fine.

import requests
from bs4 import BeautifulSoup
import pandas as pd
headers = {'User-Agent': 'Mozilla/5.0'}
for page in range(1, 2):
    response = requests.get("https://www.avbuyer.com/aircraft/private-jets/page-{page}".format(page=page),headers=headers,)

    soup = BeautifulSoup(response.content, 'html.parser')
    postings = soup.find_all('div', class_='listing-item premium')
    for post in postings:
        link = post.find('a', class_='more-info').get('href')
        link_full = 'https://www.avbuyer.com'   link
        plane = post.find('h2', class_='item-title').text
        price = post.find('div', class_='price').text
        location = post.find('div', class_='list-item-location').text
        print(location)

Output:

North America   Canada, United States - MD, For Sale by Avpro Inc.
North America   Canada, United States - WI, For Sale by Lone Mountain Aircraft Sales   
North America   Canada, United States - MD, For Sale by Avpro Inc.
North America   Canada, United States - MD, For Sale by Avpro Inc.
Europe, Monaco, For Sale by Global Jet Monaco
South America, Puerto Rico, For Sale by JetHQ
North America   Canada, United States - NE, For Sale by Duncan Aviation
North America   Canada, United States - DE, For Sale by Leading Edge Aviation Solutions
North America   Canada, United States - TX, For Sale by Par Avion Ltd.
North America   Canada, United States - MD, For Sale by Avpro Inc.
Europe, Switzerland, For Sale by Jetcraft
Europe, United Kingdom - England, For Sale by Jets4UDirect Ltd
North America   Canada, United States - MD, For Sale by Avpro Inc.
North America   Canada, United States - MT, For Sale by SkyWorld Aviation
North America   Canada, United States - MD, For Sale by Avpro Inc.
North America   Canada, United States - AZ, For Sale by Hatt & Associates
Europe, Switzerland, For Sale by Jetcraft
North America   Canada, United States - MD, For Sale by Avpro Inc.
North America   Canada, United States - MD, For Sale by Avpro Inc.
North America   Canada, United States - MD, For Sale by Avpro Inc.
  • Related