I am having trouble with getting numerical values from span tags with the same class
This is what html looks like
<ul >
<li>
<span >Comfort</span>
<span >5.0</span>
</li>
<li>
<span >Interior design</span>
<span >4.0</span>
</li>
<li>
<span >Performance</span>
<span >5.0</span>
</li>
<li>
<span >Value for the money</span>
<span >5.0</span>
</li>
<li>
<span >Exterior styling</span>
<span >5.0</span>
</li>
<li>
<span >Reliability</span>
<span >5.0</span>
</li>
</ul>
I basically want to take all the numerical values and put them in different columns, here is what I am using for my code
ua = UserAgent()
header = {'User-Agent':str(ua.safari)}
url = 'https://www.cars.com/research/nissan-leaf-2011/consumer-reviews/?page=1'
response = requests.get(url, headers=header)
print(response)
html_soup = BeautifulSoup(response.text, 'lxml')
content_list = html_soup.find_all('div', attrs={'class': 'consumer-review-container'})
data = []
for e in content_list:
data.append({
'review_title': e.h3.text,
'review_content': e.select_one('p.review-body').text,
'overall_rating': e.select_one('span.sds-rating__count').text,
'reviewer_name':e.select_one("div.review-byline div:nth-of-type(2)").text,
'review_date':e.find("div", {"class":"review-byline"}).div.text,
})
To the list data I would like to add information about: Comfort, Interior, Performance, Value for the money, Exterior styling and Reliability and this information I would like to get from the previously mentioned html code.
CodePudding user response：
I believe you are looking for something like this. add the list for display name and ratings.
data = []
for e in content_list:
data.append({
'review_title': e.h3.text,
'review_content': e.select_one('p.review-body').text,
'overall_rating': e.select_one('span.sds-rating__count').text,
'reviewer_name':e.select_one("div.review-byline div:nth-of-type(2)").text,
'review_date':e.find("div", {"class":"review-byline"}).div.text,
'rating breakdown display-name' : [d.text for d in e.select('.sds-definition-list__display-name')],
'rating breakdown value' : [d.text for d in e.select('.sds-definition-list__value')]
})
print(data)
Output:
[{'review_title': 'Great Electric Car!', 'review_content': 'This is the perfect electric car for driving around town, doing errands or even for a short daily commuter. It is very comfy and very quick. The only issue was the first gen battery. The 2011-2014 battery degraded quickly and if the owner did not have Nissan replace it, all those cars are now junk and can only go 20 miles or so on a charge. We had Nissan replace our battery with the 2nd gen battery and it is good as new!', 'overall_rating': '4.7', 'reviewer_name': 'By EVs are the future from Tucson, AZ', 'review_date': 'February 24, 2020', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['5.0', '5.0', '5.0', '5.0', '3.0', '5.0']}, {'review_title': 'Fun to drive, and no gasoline!!', 'review_content': 'For an around-the-town running errands car, this car is fabulous. Fun to drive, and super-low cost per mile (rated at over 99 mpg compared to regular gas cars).', 'overall_rating': '4.7', 'reviewer_name': 'By EV fan from Tucson, AZ', 'review_date': 'December 23, 2019', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['4.0', '4.0', '5.0', '5.0', '5.0', '5.0']}, {'review_title': 'Save movey Save Environment', 'review_content': 'It’s a a wonderful buy and I enjoy it immensely especially the fact that I don’t have to get oil changes brake changes fluids etc\nIt’s a perfect buy', 'overall_rating': '5.0', 'reviewer_name': 'By BayArea from CA', 'review_date': 'November 2, 2019', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['5.0', '5.0', '5.0', '5.0', '5.0', '5.0']}, {'review_title': 'Good money saver', 'review_content': "Car has saved me a lot of money. Electricity is much cheaper than gas, and you can't buy a car at this price, age, and mileage. \n\nThe one downside to the Leaf is the battery replacement cost is very high. Nissan raised the battery replacement cost after we bought the car, which was very frustrating.", 'overall_rating': '4.9', 'reviewer_name': 'By Lothsahn from Missouri', 'review_date': 'July 8, 2019', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['5.0', '5.0', '5.0', '5.0', '5.0', '4.0']}, {'review_title': 'What I expected', 'review_content': "There was nothing new to this leaf. I've researched them a lot and test drove one other one. This one was on par with what I've seen/drove. My mine focus was on the drive range depletion while driving. I drove 20 miles but only lost 15 miles in drive range thanks to the braking regeneration.", 'overall_rating': '3.8', 'reviewer_name': 'By brad', 'review_date': 'May 31, 2018', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['4.0', '4.0', '4.0', '4.0', '3.0', '—']}, {'review_title': "It's like driving the future!", 'review_content': "This car has been a great commuter car. It's been reliable and easy to maintain! The electric torque and power makes such a fun driving experience! Charging this car with a full charge only costs about $1, to provide up to about 70-80 miles commute. The A/C is by far the best performing of any car I have owned!", 'overall_rating': '4.4', 'reviewer_name': 'By evcarguy from Chandler, AZ', 'review_date': 'November 23, 2017', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['4.0', '4.0', '4.0', '5.0', '4.0', '5.0']}, {'review_title': 'Nice car with no cost for fuel', 'review_content': 'Nissan leaf is a nice car without any additional cost for fuel as well as maintenance for daily communication...', 'overall_rating': '4.0', 'reviewer_name': 'By Zarni Tun from Rowland Heights,CA', 'review_date': 'July 28, 2017', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['5.0', '3.0', '4.0', '5.0', '3.0', '3.0']}, {'review_title': 'No gas stations. No maintenance.', 'review_content': "I don't drive very much. My LEAF has been worry free for 6 years. NOTHING has gone wrong with it. I expect it will serve me for 20 years. I have solar panels on my roof so I drive this car for virtually $0.", 'overall_rating': '4.6', 'reviewer_name': 'By AQL from Mercer Island, WA', 'review_date': 'June 12, 2017', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['4.0', '4.0', '5.0', '5.0', '4.0', '5.0']}, {'review_title': '5 months review', 'review_content': "I got this car with 3800 miles certified pre own , 5 months ago , now at 15000 miles the cars its saving me money on gas i drive around 80-95 miles a day with no problem , you have to be careful not to be speeding around town but order that that my car will give an average of 90 miles on a charge, i drive around 2000 miles a month so the money i am saving on gas is a lo t, pretty much the car pay it self , the more you drive the more you save, i haven't lost any bars yet , i know that's coming , i will continue to use and abuse my car , until that day comes around.", 'overall_rating': '4.9', 'reviewer_name': 'By JT from kissimmee, fl', 'review_date': 'November 26, 2014', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['5.0', '5.0', '5.0', '5.0', '4.0', '5.0']}, {'review_title': 'Problems with battery capacity long-term', 'review_content': 'We have owned our Leaf since May 2011. We have loved the car but are now getting quite concerned. My husband drives the car, on average, 20-40 miles/day to and from work and running errands, mostly 100% on city roads. We live in San Diego, so no issue with winter weather and we live 7 miles from the ocean so seldom have daytime temperatures above 85. Originally, we would get 65-70 miles per 80-90% charge. Last fall we noticed that there was considerably less remaining charge left after a day of driving. He began to track daily miles, remaining "bars", as well as started charging it 100%. For 9 months we have only been getting 40-45 miles on a full charge with only 1-2 "bars" remaining at the end of the day. Sometimes it will be blinking and "talking" to us to get to a charging place ASAP. We just had it into the dealership. Though on a full charge, the car gauge shows 12 bars, the dealership states that the batteries have lost 2 bars via the computer diagnostics (which we are told is a different reading from the car gauge itself) and, that they say, is average and excepted for the car at this age. Everything else (software, diagnostics, etc.) shows 100%, so the dealership thinks that the car is functioning as it should. They are unable to explain why we can only go 40-45 miles on a charge, but keep saying that the car tests out fine. If the distance one is able to drive on a full charge decreases any further, it will begin to render the car useless. As someone else recommended, in retrospect, the best way to go is to lease the Leaf so that battery life is not an issue.', 'overall_rating': '3.7', 'reviewer_name': 'By Cathie from San Diego', 'review_date': 'July 21, 2014', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['5.0', '5.0', '2.0', '3.0', '5.0', '3.0']}]
CodePudding user response：
To get the result you could iterate over the <li> and extract the contents with .stripped_strings in a dict comprehension then update your existing dict and append it to data.
Creating a DataFrame this will create separate columns for each item:
for e in content_list:
d = {
'review_title': e.h3.text,
'review_content': e.select_one('p.review-body').text,
'overall_rating': e.select_one('span.sds-rating__count').text,
'reviewer_name':e.select_one("div.review-byline div:nth-of-type(2)").text,
'review_date':e.find("div", {"class":"review-byline"}).div.text,
}
d.update(dict(s.stripped_strings for s in e.select('ul.sds-definition-list li')))
data.append(d)
data
Output:
[{'review_title': 'Great Electric Car!',
'review_content': 'This is the perfect electric car for driving around town, doing errands or even for a short daily commuter. It is very comfy and very quick. The only issue was the first gen battery. The 2011-2014 battery degraded quickly and if the owner did not have Nissan replace it, all those cars are now junk and can only go 20 miles or so on a charge. We had Nissan replace our battery with the 2nd gen battery and it is good as new!',
'overall_rating': '4.7',
'reviewer_name': 'By EVs are the future from Tucson, AZ',
'review_date': 'February 24, 2020',
'Comfort': '5.0',
'Interior design': '5.0',
'Performance': '5.0',
'Value for the money': '5.0',
'Exterior styling': '3.0',
'Reliability': '5.0'},...]
CodePudding user response：
According to your comment,it's working.Each li tag has two child span tags and span:-soup-contains() including text value will select the first span from li and ~ will catch the immediate next tag which is the 2nd span of li tag
from bs4 import BeautifulSoup
import requests
#ua = UserAgent()
header = {'User-Agent':'mozila/5.0'}
url = 'https://www.cars.com/research/nissan-leaf-2011/consumer-reviews/?page=1'
response = requests.get(url, headers=header)
print(response)
html_soup = BeautifulSoup(response.text, 'lxml')
content_list = html_soup.find_all('div', attrs={'class': 'consumer-review-container'})
data = []
for e in content_list:
data.append({
'review_title': e.h3.text,
'review_content': e.select_one('p.review-body').text,
'overall_rating': e.select_one('span.sds-rating__count').text,
'reviewer_name':e.select_one("div.review-byline div:nth-of-type(2)").text,
'review_date':e.find("div", {"class":"review-byline"}).div.text,
'comfort':e.select_one('span:-soup-contains("Comfort")~span').text,
'Interior design':e.select_one('span:-soup-contains("Interior design")~span').text,
'Performance':e.select_one('span:-soup-contains("Performance")~span').text,
'Value for the money':e.select_one('span:-soup-contains("Value for the money")~span').text,
'Exterior styling':e.select_one('span:-soup-contains("Exterior styling")~span').text,
'Reliability':e.select_one('span:-soup-contains("Reliability")~span').text
})
print(data)
Output:
<Response [200]>
[{'review_title': 'Great Electric Car!', 'review_content': 'This is the perfect electric car for driving around town, doing errands or even for a short daily commuter. It is very comfy and very quick. The only issue was the first gen battery. The 2011-2014 battery degraded quickly
and if the owner did not have Nissan replace it, all those cars are now junk and can only go 20 miles or so on a charge. We had Nissan replace our battery with the 2nd gen battery and it is good as new!', 'overall_rating': '4.7', 'reviewer_name': 'By EVs are the future from Tucson, AZ', 'review_date': 'February 24, 2020', 'comfort': '5.0', 'Interior design': '5.0', 'Performance': '5.0', 'Value for the money': '5.0', 'Exterior styling': '3.0', 'Reliability': '5.0'},
{'review_title': 'Fun to drive, and no gasoline!!', 'review_content': 'For an around-the-town
running errands car, this car is fabulous. Fun to drive, and super-low cost per mile (rated at over 99 mpg compared to regular gas cars).', 'overall_rating': '4.7', 'reviewer_name': 'By EV fan from Tucson, AZ', 'review_date': 'December 23, 2019', 'comfort': '4.0', 'Interior design': '4.0', 'Performance': '5.0', 'Value for the money': '5.0', 'Exterior styling': '5.0', 'Reliability': '5.0'}, {'review_title': 'Save movey Save Environment', 'review_content': 'It’s a a
wonderful buy and I enjoy it immensely especially the fact that I don’t have to get oil changes brake changes fluids etc\nIt’s a perfect buy', 'overall_rating': '5.0', 'reviewer_name': 'By BayArea from CA', 'review_date': 'November 2, 2019', 'comfort': '5.0', 'Interior design': '5.0', 'Performance': '5.0', 'Value for the money': '5.0', 'Exterior styling': '5.0', 'Reliability': '5.0'}, {'review_title': 'Good money saver', 'review_content': "Car has saved me a lot of money. Electricity is much cheaper than gas, and you can't buy a car at this price, age, and mileage. \n\nThe one downside to the Leaf is the battery replacement cost is very high. Nissan raised the battery replacement cost after we bought the car, which was very frustrating.", 'overall_rating': '4.9', 'reviewer_name': 'By Lothsahn from Missouri', 'review_date': 'July
8, 2019', 'comfort': '5.0', 'Interior design': '5.0', 'Performance': '5.0', 'Value for the money': '5.0', 'Exterior styling': '5.0', 'Reliability': '4.0'}, {'review_title': 'What I expected', 'review_content': "There was nothing new to this leaf. I've researched them a lot and test drove one other one. This one was on par with what I've seen/drove. My mine focus was on the
drive range depletion while driving. I drove 20 miles but only lost 15 miles in drive range thanks to the braking regeneration.", 'overall_rating': '3.8', 'reviewer_name': 'By brad', 'review_date': 'May 31, 2018', 'comfort': '4.0', 'Interior design': '4.0', 'Performance': '4.0', 'Value for the money': '4.0', 'Exterior styling': '3.0', 'Reliability': '—'}, {'review_title': "It's like driving the future!", 'review_content': "This car has been a great commuter car. It's been reliable and easy to maintain! The electric torque and power makes such a fun driving experience! Charging this car with a full charge only costs about $1, to provide up to about
70-80 miles commute. The A/C is by far the best performing of any car I have owned!", 'overall_rating': '4.4', 'reviewer_name': 'By evcarguy from Chandler, AZ', 'review_date': 'November 23, 2017', 'comfort': '4.0', 'Interior design': '4.0', 'Performance': '4.0', 'Value for the money': '5.0', 'Exterior styling': '4.0', 'Reliability': '5.0'}, {'review_title': 'Nice car with
no cost for fuel', 'review_content': 'Nissan leaf is a nice car without any additional cost for fuel as well as maintenance for daily communication...', 'overall_rating': '4.0', 'reviewer_name': 'By Zarni Tun from Rowland Heights,CA', 'review_date': 'July 28, 2017', 'comfort': '5.0', 'Interior design': '3.0', 'Performance': '4.0', 'Value for the money': '5.0', 'Exterior styling': '3.0', 'Reliability': '3.0'}, {'review_title': 'No gas stations. No maintenance.', 'review_content': "I don't drive very much. My LEAF has been worry free for 6 years. NOTHING has gone wrong with it. I expect it will serve me for 20 years. I have solar panels on my roof
so I drive this car for virtually $0.", 'overall_rating': '4.6', 'reviewer_name': 'By AQL from Mercer Island, WA', 'review_date': 'June 12, 2017', 'comfort': '4.0', 'Interior design': '4.0', 'Performance': '5.0', 'Value for the money': '5.0', 'Exterior styling': '4.0', 'Reliability': '5.0'}, {'review_title': '5 months review', 'review_content': "I got this car with 3800 miles certified pre own , 5 months ago , now at 15000 miles the cars its saving me money on gas i drive around 80-95 miles a day with no problem , you have to be careful not to be speeding around town but order that that my car will give an average of 90 miles on a charge, i drive around 2000 miles a month so the money i am saving on gas is a lo t, pretty much the car pay it self , the more you drive the more you save, i haven't lost any bars yet , i know that's coming , i will continue to use and abuse my car , until that day comes around.", 'overall_rating':
'4.9', 'reviewer_name': 'By JT from kissimmee, fl', 'review_date': 'November 26, 2014', 'comfort': '5.0', 'Interior design': '5.0', 'Performance': '5.0', 'Value for the money': '5.0', 'Exterior styling': '4.0', 'Reliability': '5.0'}, {'review_title': 'Problems with battery capacity long-term', 'review_content': 'We have owned our Leaf since May 2011. We have loved the car but are now getting quite concerned. My husband drives the car, on average, 20-40 miles/day to
and from work and running errands, mostly 100% on city roads. We live in San Diego, so no issue with winter weather and we live 7 miles from the ocean so seldom have daytime temperatures above 85. Originally, we would get 65-70 miles per 80-90% charge. Last fall we noticed that there was considerably less remaining charge left after a day of driving. He began to track daily miles, remaining "bars", as well as started charging it 100%. For 9 months we have only been
getting 40-45 miles on a full charge with only 1-2 "bars" remaining at the end of the day. Sometimes it will be blinking and "talking" to us to get to a charging place ASAP. We just had it into the dealership. Though on a full charge, the car gauge shows 12 bars, the dealership states that the batteries have lost 2 bars via the computer diagnostics (which we are told is a different reading from the car gauge itself) and, that they say, is average and excepted for the car at this age. Everything else (software, diagnostics, etc.) shows 100%, so the dealershipcar tests out fine. If the distance one is able to drive on a full chargeain why we can only g decreases any further, it will begin to render the car useless. As someof the distance one isne else recommended, in retrospect, the best way to go is to lease the Leender the car uselessaf so that battery life is not an issue.', 'overall_rating': '3.7', 'reviase the Leaf so that ewer_name': 'By Cathie from San Diego', 'review_date': 'July 21, 2014', ' 'By Cathie from San comfort': '5.0', 'Interior design': '5.0', 'Performance': '2.0', 'Value fn': '5.0', 'Performanor the money': '3.0', 'Exterior styling': '5.0', 'Reliability': '3.0'}]