I want to webscrape the following website: 'https://sprs.parl.gov.sg/search/home'

But when my code clicks on the "Search button", I get the error:

ElementClickInterceptedException: element click intercepted: Element is not clickable at point (320, 1395)

I am not sure if it's be'cos the "Search button" is not completely visible and needs scrolling down. If so, how do I scroll down the page using Selenium? Thank you

from bs4 import BeautifulSoup as bs 
from selenium import webdriver
import time


driver = webdriver.Chrome(executable_path="C:/Users/chromedriver.exe")

page_url = 'https://sprs.parl.gov.sg/search/home'
driver.get(page_url)

# Get search box and fill it up
search = driver.find_element_by_css_selector('#divmpscreen2 > div.row > div:nth-child(1) > div > div:nth-child(1) > input')
search.send_keys('COS')

# This will select the 13th parliament
session = driver.find_element_by_xpath("//select/option[contains(text(), '13th')]")
session.click()
time.sleep(10)

# Find submit element and click
submit = driver.find_element_by_xpath("//button[@label='Search']/em")
submit.click()

CodePudding user response：

This is how you suppose to click an element using Selenium:

clickLastFoundElement():       
    try:
        builder = Actions(browser)
        builder.moveToElement(my_element).click().build().perform()
    except ElementNotInteractableException:
        try:
            my_element.click()          
        except Exception:
            hardClickElement(my_element)

hardClickElement(element):
    executor = (JavascriptExecutor)browser
    executor.executeScript('arguments[0].click();', element)

Sometimes an element won't be clicked, and you will have to "hard" click it by injecting JavaScript code into it.

CodePudding user response：

We can try click by using moving to element or by using java script executor

Actions action = new Actions(webdriver);
WebElement we = webdriver.findElement(By.xpath("path here"));
action.moveToElement(we).click().build().perform();

using javascript

WebElement element = driver.findElement(By.id("gbqfd"));
JavascriptExecutor executor = (JavascriptExecutor)driver;
executor.executeScript("arguments[0].click();", element);

CodePudding user response：

To click on the Search button you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following locator strategy:

driver.get('https://sprs.parl.gov.sg/search/home')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#divmpscreen2 > div.row > div:nth-child(1) > div > div:nth-child(1) > input"))).send_keys('COS')
Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//label[text()='By Parliament']//following-sibling::select[1]")))).select_by_value('13: 13')
driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@class='btn btn-black' and @label='Search']/em"))))

Browser Snapshot:

Note: You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

CodePudding user response：

This is one way to select the desired info and click that button (EDIT: a better way without ActionChains, and working on smaller sized windows as well):

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select
import time as t


chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument('disable-notifications')
chrome_options.add_argument("window-size=1280,720")

webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)

url = 'https://sprs.parl.gov.sg/search/home'

browser.get(url) 
searchbox = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//label[contains(text(), 'By Keyword')]/following-sibling::input")))
print(searchbox.location_once_scrolled_into_view)
searchbox.send_keys('COS')
govt_nr = Select(WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//label[contains(text(), 'By Parliament')]/following-sibling::select"))))
govt_nr.select_by_index(13) 
browser.execute_script("window.scrollTo(0,document.body.scrollHeight);")
t.sleep(1)
submit_button = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR, "button[label = 'Search']")))
submit_button.click()

Setup is linux/chromedriver, you just have to observe the imports, and the code after defining the browser/driver.

Selenium docs: https://www.selenium.dev/documentation/

One more thing to keep in mind: results will open in a new tab, so you will need to switch to it using something to the tune of switch_to.window(driver.window_handles[-1]).