I would like to load multple pages from a single website and extract specific attributes from different classes as below. Then I woule like to create a dataframe with parsed information from multiple pages.

Extract from multiple pages

for page in range(1,10):
    url = f"https://www.consilium.europa.eu/en/press/press-releases/?page={page}"
    res = requests.get(url)
    soup = bs(res.text, 'lxml')

Parsing

soup_content = soup.find_all('li', {'class':['list-item ceu clearfix','list-item gsc clearfix','list-item euco clearfix','list-item eg clearfix' ]})

datePublished = []
headline = []
description =[]
urls = []

for i in range(len(soup_content)):
    datePublished.append(soup_content[i].find('span', {'itemprop': 'datePublished'}).attrs['content'])
    headline.append(soup_content[i].find('h3', {'itemprop': 'headline'}).get_text().strip())
    description.append(soup_content[i].find('p', {'itemprop': 'description'}).get_text().strip())
    urls.append('https://www.consilium.europa.eu{}'.format(soup.find('a', {'itemprop': 'url'}).attrs['href']))

To DataFrame

df = pd.DataFrame(data = zip(datePublished, headline, description, urls), columns=['date','title', 'description', 'link'])
df

CodePudding user response：

To expand on my comments, this should work:

maxPage = 9
datePublished = []
headline = []
description =[]
urls = []

for page in range(1, maxPage 1):
    url = f"https://www.consilium.europa.eu/en/press/press-releases/?page={page}"
    res = requests.get(url)
    print(f'[page {page:>3}]', res.status_code, res.reason, 'from', res.url)
    soup = BeautifulSoup(res.content, 'lxml')

    soup_content = soup.find_all('li', {'class':['list-item ceu clearfix','list-item gsc clearfix','list-item euco clearfix','list-item eg clearfix' ]})
    for i in range(len(soup_content)):
        datePublished.append(soup_content[i].find('span', {'itemprop': 'datePublished'}).attrs['content'])
        headline.append(soup_content[i].find('h3', {'itemprop': 'headline'}).get_text().strip())
        description.append(soup_content[i].find('p', {'itemprop': 'description'}).get_text().strip())
        urls.append('https://www.consilium.europa.eu{}'.format(soup.find('a', {'itemprop': 'url'}).attrs['href']))

When I ran it, 179 unique rows were collected [20 rows from all pages except the 7th, which had 19].

CodePudding user response：

There are different ways to get your goal:

• @Driftr95 comes up with a modification of yours using range(), that is fine, while iterating a specific number of pages.

• Using a while-loop to be flexible in number of pages, without knowing the exact one. You can also use a counter if you like to break the loop at a certain number of iterations.

• ...

I would recommend the second one and also to avoid the bunch of lists cause you have to ensure they have the same lenght. Instead use a single list with dicts that looks more structured.

Example

import requests
import pandas as pd
from bs4 import BeautifulSoup
base_url = 'https://www.consilium.europa.eu'
path ='/en/press/press-releases'
url = base_url path

data = []

while True:
    print(url)
    soup = BeautifulSoup(requests.get(url).text)
    for e in soup.select('li.list-item'):
        data.append({
            'date':e.find_previous('h2').text,
            'title':e.h3.text,
            'desc':e.p.text,
            'url':base_url e.h3.a.get('href')
        })

    if soup.select_one('li[aria-label="Go to the next page"] a[href]'):
        url = base_url path soup.select_one('li[aria-label="Go to the next page"] a[href]').get('href')
    else:
        break

df = pd.DataFrame(data)

Output

...