How to avoid AttributeError: 'NoneType' object has no attribute 'text' webscrapi-CodePudding

Trying to make a program that gets a random subreddit, displays the title to the terminal, asks the user if they want to see it and if so opens a browser with that subreddit.

I'm running into an issue here whilst doing so. I am very new with python/beautiful soup just started using it a few weeks ago for school any help will be very appreciated.

import requests
from bs4 import BeautifulSoup
import webbrowser

while True:
    url = requests.get("https://www.reddit.com/r/random")
    soup = BeautifulSoup(url.content, "html.parser")
    title = soup.find(class_="_2yYPPW47QxD4lFQTKpfpLQ").text ## this is supposed to get the title of the subreddit and this is where my error is occurring

    print(f"{title} \nSelect this subreddit? (Y/N)")
    ans = input("").lower()

    if ans == "y":
        url = "https://www.reddit.com/%s" % title ## Some issue, not sure what
        webbrowser.open(url)
        break
    elif ans == "n":
        print("Try again!")
        continue
    else:
        print("Wrong choice!")
        break

CodePudding user response：

soup.find(class_="_2yYPPW47QxD4lFQTKpfpLQ") returns None if there is no element with the given class, That's why you got this error

Try to catch the error using try-except and print something if soup not found any title.

Try this code.

from bs4 import BeautifulSoup

import webbrowser

while True:
    url = requests.get("https://www.reddit.com/r/random")
    soup = BeautifulSoup(url.content, "html.parser")
    try:
        title = soup.find(class_="_2yYPPW47QxD4lFQTKpfpLQ").text
    ## this is supposed to get the title of the subreddit and this is where my error is occurring
    except AttributeError:
        print("Title Not Found")

    else:
        print(f"{title} \nSelect this subreddit? (Y/N)")
        ans = input("").lower()

        if ans == "y":
            url = "https://www.reddit.com/%s" % title ## Some issue, not sure what
            webbrowser.open(url)
            break
        elif ans == "n":
            print("Try again!")
            continue
        else:
            print("Wrong choice!")
            break

CodePudding user response：

Element you try to find() is not available in your soup and calling a methode on None raises this error, so:

Always and at first check your response, does it include the expected information - You will be detected as bot, so add some headers:
```
url = requests.get('https://www.reddit.com/r/random', headers = {'User-Agent': 'Mozilla/5.0'})
```
Avoid to select your elements by dynamic looking classes, instead us static identifier and HTML structure, also select the right elements:
```
soup.h2.get_text(strip=True)
```

Example

from bs4 import BeautifulSoup
import requests, webbrowser

while True:
    url = requests.get('https://www.reddit.com/r/random', headers = {'User-Agent': 'Mozilla/5.0'})
    soup = BeautifulSoup(url.content)
    title = soup.h2.get_text(strip=True)
    print(f"{title} \nSelect this subreddit? (Y/N)")
    ans = input("").lower()

    if ans == "y":
        url = f"https://www.reddit.com/{title}"
        webbrowser.open(url)
        break
    elif ans == "n":
        print("Try again!")
        continue
    else:
        print("Wrong choice!")
        break