href inside "Load more" button doesn't bring more articles when pasting URL-CodePudding

The

The thing is, if I enter this into my browser, I get the same news I get if I just call the original website. Because of this, the only way I'm seeing I would be able to scrape the website is to create a script that recursively clicks. The thing is I need news until 2019, so it doesn't seem very feasible.

Also, when checking the event listeners I see this:

But I'm not sure how can I use that to my advantage.

Am I missing something? Is there any way to access older news through a link (or an API would be even better, but I didn't find any call to an API).

I'm currently using Python to scrape, but I'm in the investigation stage, so there's no code to show that's meaningful. Thanks a lot!

CodePudding user response：

Chech Query String format @ wiki, please.

You missing a & mark

CodePudding user response：

Unfortunately, it doesn't seem like there's a nice api or other way to get the data in bulk. You will need to iterate through the "pages"/"load more" and then parse the html.

Will take a little time, but this code will do that for you.

import requests from bs4 import BeautifulSoup import re import json from dateutil import parser import datetime import pandas as pd headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'} rows = [] endYear = 2019 continuneLoop = True page = 1 while continuneLoop == True: url = f'https://noticias.caracoltv.com/colombia?00000172-8578-d277-a9f3-f77bc3df0000-page={page}' response = requests.get(url, headers=headers) soup = BeautifulSoup(response.text, 'html.parser') if page == 1: mainArticle = soup.find('div', {'class':'ListU'}) jsonStr = str(mainArticle.find('script')) jsonStr = re.search('({.*})', jsonStr).group(1) jsonData = json.loads(jsonStr) articlePublished = jsonData['datePublished'] dt = parser.parse(articlePublished) print(f"{dt.day}-{dt.month}-{dt.year} : {jsonData['headline']}") rows.append(jsonData) subArticles = soup.find_all('li', {'class':'ListE-items-item'}) for subArticle in subArticles: jsonStr = str(subArticle.find('script')) jsonStr = re.search('({.*})', jsonStr).group(1) jsonData = json.loads(jsonStr) articlePublished = jsonData['datePublished'] dt = parser.parse(articlePublished) print(f"{dt.day}-{dt.month}-{dt.year} : {jsonData['headline']}") rows.append(jsonData) #get bottom articles loadMore = soup.find('ps-list-loadmore', {'class':'ListD'}) articles = loadMore.find_all('li',{'class':'ListD-items-item'}) for article in articles: try: jsonStr = str(article.find('script')) jsonStr = re.search('({.*})', jsonStr).group(1) jsonData = json.loads(jsonStr) articlePublished = jsonData['datePublished'] dt = parser.parse(articlePublished) except: headline = article.find('a', href=True)['title'] articlePublished = article.find('div', {'PromoB-timestamp'})['data-timestamp'] url = article.find('a', href=True)['href'] jsonData = { 'headline':headline, 'datePublished':articlePublished, 'url':url} if endYear == dt.year: print('Done') continueLoop = False else: print(f"{dt.day}-{dt.month}-{dt.year} : {jsonData['headline']}") rows.append(jsonData) page = 1 df = pd.DataFrame(rows)

Output: