End while-loop when last page is reached using beautifulsoup-CodePudding

I am trying to scrape trade publications from this site: https://www.webwire.com/IndustryList.asp

I can scrape through each individual industry section (e.g., 'Airlines / Aviation' or 'Automotive') without issue, but my loop gets stuck when it reaches the last page of results, and doesn't proceed to the next industry in the for-loop.

I don't think I'm getting an exception either, so how do I end the loop when it reaches the last available page so that it proceeds to the next item in the for-loop?

import requests
from bs4 import BeautifulSoup

industries = ["AIR","AUT","LEI"]

for industry in industries:
    print(industry)
    print("==================")
    num = 1
    while True:
        url = f"https://www.webwire.com/TradePublications.asp?ind={industry}&curpage={num}"
        page = requests.get(url)
        soup = BeautifulSoup(page.content, 'html.parser')

        for e in soup.select('#syndication-list li'):
            print(e.get_text())

        num = num   1
    else:
        break

CodePudding user response：

This example will iterate over the list industries and get all pages until the last one:

import requests
from bs4 import BeautifulSoup


industries = ["AIR", "AUT", "LEI"]
url = "https://www.webwire.com/TradePublications.asp?ind={}&curpage={}"

for ind in industries:
    u = url.format(ind, 1)
    while True:
        soup = BeautifulSoup(requests.get(u).content, "html.parser")
        for li in soup.select("#syndication-list li"):
            print("{:<10} {}".format(ind, li.text))

        next_page = soup.select_one('a:-soup-contains("Next »")')
        if next_page:
            u = (
                "https://www.webwire.com/TradePublications.asp"
                  next_page["href"]
            )
        else:
            break

Prints:


...

LEI        Women's Wear Daily/Fairchild Financial
LEI        Worcester Quarterly Magazine
LEI        Word Association/Econoguide Travel Books
LEI        Worldwide Spa Review
LEI        Worth Magazine
LEI        Y Not Girl Magazine
LEI        Yankee Driver
LEI        Ziff Davis Media

CodePudding user response：

You can make the pagination using for loop and range function as follows:

import requests
from bs4 import BeautifulSoup

industries = ["AIR","AUT","LEI"]

for industry in industries:
    # print(industry)
    # print("==================")
  
    #url = f"https://www.webwire.com/TradePublications.asp?ind={industry}&curpage=1"

    
    #print(url)
    for page in range(1,14):
        print(page)
        url=f'https://www.webwire.com/TradePublications.asp?ind={industry}&curpage={page}'
        page = requests.get(url)
        soup = BeautifulSoup(page.content, 'html.parser')
        for e in soup.select('#syndication-list li'):
            print(e.get_text())