Question:
How can I show all reviews by clicking "Show more reviews" button?
What did I do:
- To scrape all reviews, I decided to Keep clicking until that button disappears.
- But some new reviews didn't appear after clicking for 8 times (using while below).
- I already checked xpath of the button but it didn't changed.
- If I click the button manually without the driver, I can see new reviews.
import time, random
from selenium import webdriver
from latest_user_agents import get_random_user_agent
### ser user_agent
user_agent = get_random_user_agent()
options = webdriver.ChromeOptions()
options.add_argument("--user-agent=" user_agent)
driverpath = "C:/Users/~~/chromedriver.exe"
driver = webdriver.Chrome(chrome_options=options,executable_path=driverpath)
### target url
url = "https://www.sony.co.uk/electronics/truly-wireless/wf-l900/reviews-ratings"
### open URL
driver.get(url)
time.sleep(5)
# accept cookies
driver.find_element_by_xpath('//*[@id="onetrust-accept-btn-handler"]').click()
# show all reviews
try:
while True:
driver.execute_script('window.scroll(0,1000000);')
driver.find_element_by_xpath('//*[@id="reviews_listing_278405943492055451029662"]/div[3]/div[4]/button').click()
time.sleep(random.randrange(2, 5, 1))
except:
print("---------------- finish showing all reviews ----------------")
CodePudding user response:
Try going to the end of the webpage and then do it as the button might not been loaded fully.
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
You might also try to press the button only when it is available.
element = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.ID, "myElement")))
CodePudding user response:
You can try javascript click:
element = driver.find_element_by_xpath('//*[@id="reviews_listing_278405943492055451029662"]/div[3]/div[4]/button')
driver.execute_script("arguments[0].click();", element)
In this method, the scrolling shouldn't be necessary. See more info about the subject here.