Selenium: click the button element to load then load more reviews-CodePudding

Hello I am trying to download reviews from this website: https://lectricebikes.com/pages/reviews. It only shows several reviews, but if you click button "Load More Reviews" it can show more reviews. So I am thinking if i run EC.element_to_be_clickable several times, then i can crawl all reviews by EC.presence_of_all_elements_located, But it only finds the same elements before click the button. Can somebody help me out.

browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)
url = 'https://lectricebikes.com/pages/reviews' 
browser.get(url)

browser.execute_script("window.scrollTo(0,document.body.scrollHeight);")
try:
    WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="stamped-reviews-tab"]/ul/li[14]/a'))).click()
    print('button found')
except:
    print('no buttons')

reviews= WebDriverWait(browser, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'stamped-review-content-body')))
list1=[]
for r in reviews:
    list1.append(r.text)
    print(r.text)

CodePudding user response：

I think you should proceed another way :

What if you click on the show more button each second in a while loop with 1 sec delay during 2 minutes and then get all the content ?

I looked how it's made and seems to be the best solution to me.

Hope it helps you !

CodePudding user response：

Use wait time between each click

from random import randint
from time import sleep
for i in range(1,9999):
    try:
        WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="stamped-reviews-tab"]/ul/li[14]/a'))).click()
        print('button found')
        sleep(randint(7,13))
    except:
        print('no buttons')
        break