Python BeautifulSoup4 news scraper giving odd error-CodePudding

I am trying to make a news scraper with BS4 and I am able to get the html code from the website (cnn) and this is my code:

from bs4 import BeautifulSoup
import requests

url = "https://www.cnn.com/"
topic = input("What kind of news are you looking for")
result = requests.get(url)
doc = BeautifulSoup(result.text, "html.parser")

prices = doc.find_all(text = f"{topic}")
parent = prices[0].parent
print(parent)

but its giving me this error

xxx@xxx-xxx xxx % python3 news_scraper.py
What kind of news are you looking for?Coronavirus
Traceback (most recent call last):
  File "/xxx/xxx/xxx/xxx/news_scraper.py", line 10, in <module>
    parent = prices[0].parent
IndexError: list index out of range

I have no idea what is causing this, Thanks!

CodePudding user response：

If the string topic is not found on the page, then prices will be an empty array. To fix this, first check that the length of prices is not zero. Like this:

from bs4 import BeautifulSoup
import requests

url = "https://www.cnn.com/"
topic = input("What kind of news are you looking for")
result = requests.get(url)
doc = BeautifulSoup(result.text, "html.parser")
prices = doc.find_all(text = f"{topic}")

if len(prices) != 0:
     parent = prices[0].parent
     print(parent)
else:
     print("No news of that topic was found.");

CodePudding user response：

I think the issue is that most of CNN's page is dynamic and beautifulsoup can't read dynamically generated content. The sections at the bottom of the page are native to that url and it works just fine on those. You need something like Selenium for dynamic pages.

CodePudding user response：

doc = BeautifulSoup(result.text, "html.parser")
print(doc)

Add print and look there not DOM, only JS and CSS