How to scroll down google job page using selenium python-CodePudding

I am trying to scroll down the job posts using below lines, but it will give sometime correct results to scroll down to the end and sometimes it won't.

html = driver.find_element_by_tag_name('html')
time.sleep(5)
html.send_keys(Keys.END)

Can anyone suggest me how to scroll down to the end, please find the link and screenshot below.

https://www.google.com/search?q=upsc jobs in india&rlz=1C1CHBF_enIN869IN869&oq=upsc jo&aqs=chrome.1.69i57j0i433i512j0i131i433i512j0i512l3j0i131i433i512l2j0i512j0i433i512&sourceid=chrome&ie=UTF-8&ibp=htl;jobs&sa=X&sqi=2&ved=2ahUKEwjR27GN_qPzAhX4ppUCHb_0B_QQkd0GegQIORAB#fpstate=tldetail&sxsrf=AOaemvIxuJXh3if0tw7ezZfjkXRe5DSxsA:1632911697417&htivrt=jobs&htidocid=N1_BNfzt8n8auXjGAAAAAA==

CodePudding user response：

The more you scroll the more data you get, basically it's a dynamic web site. I have hardcoded 50 as a dummy number, you can have 100 or any other number for this matter.

You can use the sample code :

driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.implicitly_wait(30)
driver.get("https://www.google.com/search?q=upsc jobs in india&rlz=1C1CHBF_enIN869IN869&oq=upsc jo&aqs=chrome.1.69i57j0i433i512j0i131i433i512j0i512l3j0i131i433i512l2j0i512j0i433i512&sourceid=chrome&ie=UTF-8&ibp=htl;jobs&sa=X&sqi=2&ved=2ahUKEwjR27GN_qPzAhX4ppUCHb_0B_QQkd0GegQIORAB#fpstate=tldetail&sxsrf=AOaemvIxuJXh3if0tw7ezZfjkXRe5DSxsA:1632911697417&htivrt=jobs&htidocid=N1_BNfzt8n8auXjGAAAAAA==")
j = 1
for i in range(50):
    element = driver.find_element(By.XPATH, f"(//div[@role='heading'])[{j}]")
    driver.execute_script("arguments[0].scrollIntoView(true);", element)
    j =  j   1

CodePudding user response：

You can also try this to scroll till the end.

driver.get("https://www.google.com/search?q=upsc jobs in india&rlz=1C1CHBF_enIN869IN869&oq=upsc jo&aqs=chrome.1.69i57j0i433i512j0i131i433i512j0i512l3j0i131i433i512l2j0i512j0i433i512&sourceid=chrome&ie=UTF-8&ibp=htl;jobs&sa=X&sqi=2&ved=2ahUKEwjR27GN_qPzAhX4ppUCHb_0B_QQkd0GegQIORAB#fpstate=tldetail&sxsrf=AOaemvIxuJXh3if0tw7ezZfjkXRe5DSxsA:1632911697417&htivrt=jobs&htidocid=N1_BNfzt8n8auXjGAAAAAA==")
i = 0
try:
    while True:
        options = driver.find_elements_by_xpath("//div[@role='treeitem']")
        driver.execute_script("arguments[0].scrollIntoView(true);",options[i])
        i =1
        time.sleep(.5)
except:
    pass