I am trying to extract customer reviews from: https://www.booking.com/hotel/sg/great-world-service-apartments.html#tab-reviews
The website currently has 6 pages containing reviews. Using beautiful soup and Selenium, I only managed to scrape the first page of reviews as the URL does not seem to change for each page. However, under request, the difference between each new page lies in the offset value ( 10 for each new page). For example,
So far, I found the page number list too as attached below (Under Inspect). Would it be advisable to make use of page number list or the difference in offset values to perform the scraping for the rest of the pages?
My current codes are as follows:
import scrapy
import pandas as pd
import numpy as np
import re
import requests
import bs4
from bs4 import BeautifulSoup
import csv
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
#Install driver
driver = webdriver.Chrome('./chromedriver.exe')
#specify URL
url1 = 'https://www.booking.com/hotel/sg/great-world-service-apartments.html#tab-reviews'
driver.get(url1)
time.sleep(5)
html = driver.page_source
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36'}
r= requests.get(url1,{'headers':headers})
soup = bs4.BeautifulSoup(r.text,'html.parser')
title = soup.head.title
print(title)
print(list(soup.children))
print(list(soup.parents))
titleparent = soup.title.parent.name
print(titleparent)
#Get all reviews
from selenium.webdriver.common.by import By
reviews = driver.find_element(by=By.CLASS_NAME, value='review_list_new_item_block')
print(reviews)
#reviews = soup.find(class_='review_list')
#title = reviews['title']
#reviews = soup.find_all('ul',class_='review_list')
#print(title)
#Find attributes
#1) Customer Name
names = soup.find_all('span',class_='bui-avatar-block__title')
#print(names)
cust_names = []
for i in range(0,len(names)):
cust_names.append(names[i].get_text())
cust_names = cust_names[:10]
print(cust_names)
#2) Country
country = soup.find_all('span',class_='bui-avatar-block__subtitle')
#print(country)
country_list = []
for i in range(0,len(country)):
country_list.append(country[i].get_text())
country_list[:] = [country.strip('\n') for country in country_list]
country_list = country_list[:10]
print(country_list)
#3) Header
header = soup.find_all('div',string=lambda text:"bui-grid__column-9 c-review-block__right")
#print(header)
headers_list = []
for i in range(0,len(header)):
headers_list.append(header[i].get_text())
headers_list[:] = [header.strip('\n') for header in headers_list]
headers_list = headers_list[:10]
#print(headers_list)
#4) Positive Reviews
positive_reviews = soup.find_all('span',{'class':'c-review__body'},{'svg':'bk-icon -iconset-review_great c-review__icon'})
#print(positive_reviews)
pos_reviews_list = []
for i in range(0,len(positive_reviews)):
pos_reviews_list.append(positive_reviews[i].get_text())
#pos_reviews_list[:] = [positive_reviews.strip('\n') for positive_reviews in pos_reviews_list]
print(pos_reviews_list)
#Store into a dataframe
data = {'Customer Name': cust_names,
'Country': country_list,'Positive Reviews':pos_reviews_list}
GWSAReviews1 = pd.DataFrame(data=data)
print(GWSAReviews1)
#GWSAReviews1.to_csv('GWSAReviews1.csv')Would appreciate if someone can help me on this, thank you in advance!
CodePudding user response：
I cannot add comment as i dont have enough reputations. but i believe its not only the offset which is changing, the other part of url is also changing
=1650936333328
=1650936333332
=1650936333338
please check if this affects the way you want to write code, its just an observation and suggestion.
CodePudding user response：
There is no problem with your URLs , you can use them and you can limited page number list to solve your problem as mentioned