Scraping reviews from product page using python-CodePudding

'''I'm writing this code but it is only giving me top 4 reviews. I want all the review's of a product from product page. Please help me out with my code..'''

url = 'https://mamaearth.in/product/glow-serum-foundation-almond-glow'
r = requests.get(url)
soup=BeautifulSoup(r.content,'lxml')
pro = html.fromstring(r.content)
driver.maximize_window()
driver.implicitly_wait(20)
button2 = driver.find_element_by_xpath('//div[@style="color: rgb(0, 174, 239); cursor: `pointer;"]')
driver.execute_script("arguments[0].click();", button2)
review = pro.xpath('//div[@]/text()')
reviewer_details = pro.xpath('//span[@]/text()')

CodePudding user response：

You can access the data in the json and return a list of dictionaries, each containing the review information:

import requests
from bs4 import BeautifulSoup
import json

url = 'https://mamaearth.in/product/glow-serum-foundation-almond-glow'
r = requests.get(url)
soup=BeautifulSoup(r.content,'lxml')

data = json.loads(soup.find('script', type='application/ld json').text)
reviews = data['review']

An example of the output:

{'@type': 'Review', 'datePublished': '2022-06-21 11:31:40', 'reviewBody': 'Great product', 'author': {'@type': 'Person', 'name': 'Tahamira khatun'}}
{'@type': 'Review', 'datePublished': '2022-06-17 14:18:01', 'reviewBody': 'Very nice for products.... I like ', 'author': {'@type': 'Person', 'name': 'Uma'}}
{'@type': 'Review', 'datePublished': '2022-06-08 17:16:03', 'reviewBody': "It's super awesome gives instant glow ✨️ ", 'author': {'@type': 'Person', 'name': 'Swati '}}
{'@type': 'Review', 'datePublished': '2022-06-06 15:59:25', 'reviewBody': 'Really great product ♥️', 'author': {'@type': 'Person', 'name': 'Bincy'}}
{'@type': 'Review', 'datePublished': '2022-06-04 09:06:03', 'reviewBody': 'Awesome', 'author': {'@type': 'Person', 'name': 'Kareema'}}
{'@type': 'Review', 'datePublished': '2022-06-01 06:12:54', 'reviewBody': 'Nice  product ', 'author': {'@type': 'Person', 'name': 'Tiki mishra '}}
{'@type': 'Review', 'datePublished': '2022-05-31 19:14:13', 'reviewBody': 'I m happy Than you', 'author': {'@type': 'Person', 'name': 'Naziya'}}
{'@type': 'Review', 'datePublished': '2022-05-21 18:10:39', 'reviewBody': 'Ubtan face wash ', 'author': {'@type': 'Person', 'name': 'Love you mama earth ❤️'}}