Beautifulsoup select an element based on the innerHTML with Python-CodePudding

I am writing a python script with Beautifulsoup to scrap the questions page on Stack Overflow. I only want to get all of the question titles along with the votes from the page.

For each of the questions, I get 3 div elements(votes, answers, views) using the same class. It looks something like this:

<div >
    <div >
        <span >0</span>
        <span >votes</span>
    </div>
    <div >
        <span >10</span>
        <span >answer</span>
    </div>
    <div >
        <span >15</span>
        <span >views</span>
    </div>
</div>

My python code looks like this

from tkinter import DoubleVar
from urllib import response
import requests
from bs4 import BeautifulSoup

url = "https://stackoverflow.com/questions"
response = requests.get(url)

soup = BeautifulSoup(response.text, "html.parser")
questions = soup.select(".s-post-summary.js-post-summary")

for question in questions:
    print(question.select_one(".question").getText())
    # Need help select the votes
    print(question.select_one(".item-number").getText())

Since votes, answers and views all share the same classes, what is the best way and the least fragile way to only get the votes count?

CodePudding user response：

It's working

import requests
from bs4 import BeautifulSoup

url = "https://stackoverflow.com/questions"
response = requests.get(url)

soup = BeautifulSoup(response.text, "html.parser")

title = [x.get_text(strip=True) for x in soup.select('[] > a')]
print(title)
votes =  [x.get_text(strip=True) for x in soup.select('div[] > span:nth-child(1)')]
print(votes)

Output:

['React Native - expo/vector-icons typescript type definition for icon name', 'React 25 5 Clock is working but fails all tests', 'Add weekly tasks, monthly tasks in google spreadsheet', 'Count number of change in values in Pandas column', "React-Select: How do I update the selected option dropdown's defaultValue on selected value onChange?", 'Block execution over a variable (TTS Use-Case), other than log statements (spooky)', "'npm install firebase' hangs in wsl. runs fine in windows", 'Kubernetes Dns service sometimes not working', 'Neo4j similarity of single node with entire graph', 'What is this error message? ORA-00932: inconsistent datatypes: expected DATE got NUMBER', 'Why getChildrenQueryBuilder of NestedTreeRepository say Too few parameters: the query defines 2 parameters but you only bound 0', 'Is is a security issue that Paypal uses dynamic certificate to verify webhook notification?', 'MessageBox to autoclose after 
a function done', 'Can someone clearly explain how this function is working?', 'Free open-sourced tools for obfuscating iOS app?', "GitHub page is not showing background image, FF console 
shows couldn't load images", 'Is possible to build a MLP model with the tidymodels framework?', 'How do I embed an interactive Tableau visual into an R Markdown script/notebook on Kaggle?', 'Dimensionality reduction methods for data including categorical variables', 'Reaching localhost api from hosted static site', 'Finding the zeros of a two term exponential function with 
python', 'optimizing synapse delta lake table not reducing the number of files', '(GAS) Email 
Spreadsheet range based on date input in the cell', 'EXCEL Formula to find and copy cell based on criteria', 'how to write function reduce_dimensionality?', 'Semi-Radial type Volume Slider in WPF C#', 'tippy.js tool tips stop working after "window.reload()"', 'is there some slice indices must be integers on FFT opencv python? because i think my coding is correct', 'NoParameterFoundException', 'How to get the two Input control elements look exactly same in terms of background and border?', 'My code is wrong because it requests more data than necessary, how can i solve it?', 'Express Session Not Saving', 'Which value should I search for when changing the date by dragging in FullCalendar?', 'Non-constant expression specified where only constant 
expressions are allowed', 'Cocoapods not updating even after latest version is installed', 'Ruby "Each with Index" starting at 1?', 'Converting images to Pytorch tensors loses label data', 'itemview in Adapter for recyclerview not getting id from xml', 'Use Margin Auto & Flex to Align Text', '(C  ) URLDownloadToFile Function corrupting downloaded EXE', 'Search plugin for Woocommerce website (Free)', 'Create new folder when save image in Python Plotly', "What's the difference between avfilter_graph_parse_ptr() and avfilter_link()?", 'Inputs to toString (java) on a resultset from MySQL [duplicate]', 'Which language i learn in This time for better future? python or javaScript? [closed]', 'Hi everyone. I want to write a function in python for attached data frame. I can not figure out how can I do it', 'is there a way in R to mutate a cumulative subtraction to update the same mutated var?', 'making a simple reccommendation system in JavaScript', 'Amchart4 cursor does not match mouse position in screen with zoom', 'Bash curl command works in terminal, but not with Python os.system()']
['0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '-2', '0', '1', '0', '0', '0']

CodePudding user response：

you can use find_all() function, and the return is an array. because of the votes in index 0, you can access in index 0.

print(question.find_all(".item-number")[0].getText())

CodePudding user response：

You can select for a shared parent element that returns a list of all the questions then use select_one and a css class selector pick up the vote and question title. Wrap this in a dictionary comprehension to pair results up.

import requests
from bs4 import BeautifulSoup as bs

r = requests.get("https://stackoverflow.com/questions")
soup = bs(r.content, "lxml")
data = {
    i.select_one(".s-post-summary--content-title").text.strip(): int(
        i.select_one(".s-post-summary--stats-item-number").text
    )
    for i in soup.select(".s-post-summary")
}
print(data)