Why does the BeautifulSoup select() method return empty list?-CodePudding

import requests
from bs4 import BeautifulSoup


response = requests.get('https://stackoverflow.com/questions')
soup = BeautifulSoup(response.text, 'html.parser')

questions = soup.select('.question-summary')
print(questions)

This returns:

[]

From the information in the https://codewithmosh.com/courses/ python course I payed for, this should not had happened.

Why does this code return [[]]?

CodePudding user response：

maybe you must add any params in your requests

CodePudding user response：

I inspected the source code of https://stackoverflow.com/questions, and noticed there's no class question-summary used.

But there's a s-post-summary and a s-post-summary--stats-item class, have you tried them?

CodePudding user response：

Your code returns [] because there is no element with the class .question-summary

You should always inspect the website first. As FLAK-ZOSO said, I also couldn't find any question-summary class in the HTML.

You can get the question titles using

question_titles = soup.select('.s-post-summary--content-title')

and question summaries by

question_summaries = soup.select('.s-post-summary--content-excerpt')

Example:

import requests
from bs4 import BeautifulSoup


response = requests.get('https://stackoverflow.com/questions')
soup = BeautifulSoup(response.text, 'html.parser')

question_summaries = soup.select('.s-post-summary--content-excerpt')
print(question_summaries[0].text.strip())