BeautifulSoup Returns empty list which leads to an IndexError in my Python code-CodePudding

I am trying to do web scraping using BeautifulSoup. The code I have written is below:

import requests
from bs4 import BeautifulSoup

response = requests.get("https://stackoverflow.com/questions")
soup = BeautifulSoup(response.text, "html.parser")

questions = soup.select(".question-summary")
print(type(questions[0]))

When I run the code, I get the error message below:

print(type(questions[10]))
IndexError: list index out of range

Then i tried to print the list like below:

import requests
from bs4 import BeautifulSoup

response = requests.get("https://stackoverflow.com/questions")
soup = BeautifulSoup(response.text, "html.parser")

questions = soup.select(".question-summary")
print(questions)

And then I get an empty list: []

What am I doing wrong?

Thanks for your answers.

CodePudding user response：

.question-summary is incorrect locator because it's a portion of id meaning each id value start with question-summary. Now it's working.

import requests
from bs4 import BeautifulSoup

response = requests.get("https://stackoverflow.com/questions")
soup = BeautifulSoup(response.text, "html.parser")

questions = soup.select('[id^="question-summary"]')
print(questions)

Output:

1" data-post-type-id="1" id="question-summary-71715531">
<div >
<div  title="Score of 0">
<span >0</span>
<span >votes</span>
</div>
<div  title="0 answers">
<span >0</span>
<span >answers</span>
</div>
<div  title="5 views">
<span >5</span>
<span >views</span>
</div>
</div>
<div >
<h3 >
<a  href="/questions/71715531/is-it-possible-to-draw-a-logistic-regression-graph-with-multiple-x-variable">Is it possible to draw a 
logistic regression graph with multiple x variable?</a>
</h3>
<div >
                Currently, this is my X and V value. May I know is it possible to draw a logistic regression curve with X that has multiple column? Or I am required to draw multiple graphs to do so?
X = df1.drop(['...
            </div>
<div >
<div >
<a  href="/questions/tagged/python-3.x" rel="tag" title="show questions tagged 'python-3.x'">python-3.x</a> <a  href="/questions/tagged/machine-learning" rel="tag" title="show questions tagged 'machine-learning'">machine-learning</a>
</div>
<div >
<a  href="/users/14128881/christopher-chua"> <div  data-user-id="14128881">
<img ,="" alt="user avatar"  height="16" src="https://lh6.googleusercontent.com/-Sn3B_E5hiJc/AAAAAAAAAAI/AAAAAAAAAAA/AMZuucl1oyfdhJiXhrx73JLYqzKAK9icag/photo.jpg?sz=32" width="16"/>
</div>
</a>
<div >
<div >
<a  href="/users/14128881/christopher-chua">Christopher Chua</a>
</div>
<ul >
<li ><span  dir="ltr" title="reputation score ">7</span></li>
</ul>
</div>
<time >asked <span  title="2022-04-02 07:03:06Z">13 mins ago</span></time>

.. so on