I used selenium for scraping, but out of 56 listing i can get only 40 listings.but the class name is-CodePudding

main url :https://www.kaplanpathways.com/degree-finder/#/search-result?status=7&institution_short_name=Arizona-State-University-Downtown-Phoenix-Campus&subject_area_name=&university=38&degree_level=20

try:

for i in range(1, 20):

    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    driver.find_element_by_xpath("/html/body/div[5]/main/div/div/app-root/app-search-result/div/div[2]/div[2]/div[2]/button[1]").click()

except: pass

course_list = driver.find_elements_by_xpath("//*[@class='wrap-result']")

print("Total courses: ", len(course_list))

CodePudding user response：

Problem: When you are moving to the bottom of the page, you actually move out of the clickable area for the "Show more" button. selenium does not click the button if that is not clickable (i.e. out of the screen, or behind some div like accept cookies div in your case).

Solution: Try clicking by injecting javascript.

driver.execute_script("""document.querySelector(".dgf-show-more-button").click()""")

Note: Also don't forgot to click the "I accept" button for cookies.