Why search retrieve no result? with selenium-CodePudding

I have been trying to scrape this site and sending in the search phrase cause error.

AttributeError: 'NoneType' object has no attribute 'send_keys'

I have searched StackOverflow and the error usually is resolved when you redefine the web element again which I did but still getting the same error.

I do not want to directly go to the search link like https://www.couriermail.com.au/search-results?q=Anthony Albanese instead I want to click on the search icon and after the input field is open send in the search phrase.

Here is what I have tried

            WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "/html/body/nav/form/button"))).click() 
            search = driver.find_element(by=By.XPATH,value='/html/body/nav/form/button').click()

            search.send_keys("Anthony Albanese")
            time.sleep(10)
            search.send_keys(Keys.ENTER)

I have tried simulating the typing-like behavior but it results in other errors stackoverflow help

This is the HTML of the page that I am referring to The HTML of the search icon so my question is what I am doing wrong? I have tried redefining the web element or imitating the typing-like behavior but I am not successful in getting the URLs of articles

CodePudding user response：

Yet another answer, but with relative xpaths instead of absolute. Also, instead of simulating ENTER key, I have used the clicking of search button.

driver.get("https://www.couriermail.com.au/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[@class='navigation']//button[@name='Search']"))).click()
input_text = driver.find_element(by=By.XPATH, value="//form[contains(@class, 'navigation_search')]//input")
input_text.send_keys("Anthony Albanese")
print(input_text.get_attribute('value'))
time.sleep(2)
driver.find_element(By.XPATH, "//*[@class='navigation']//button[@name='Search']").click()
time.sleep(20)
# WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.XPATH, "//*[@class='storyblock_title']")))
all_articles = driver.find_elements(By.XPATH, "//*[@class='storyblock_title']")
article_headers = [article.text for article in all_articles]
print(article_headers)
print(driver.current_url)
assert driver.current_url == "https://www.couriermail.com.au/search-results?q=Anthony Albanese"
driver.quit()

Output:

   Anthony Albanese
['Albo saves Cutters’ PNG fixture after boycott threat', 'Sport diplomacy: Why a QRL game is so important to the PM', 'Albo’s cheeky dig at ScoMo during Midnight Oil event', 'How Albo amassed huge property portfolio', '‘I can’t solve everything’: PM’s big admission', 'PM admits: ‘Can’t solve everything’', "Anthony Albanese 'emulated Bob Hawke'", "People 'cheered like idiots' at Anthony Albanese skolling beer", "'Hold my beer': Anthony Albanese 'trying to be as popular as Hawke'", 'I want to look at ‘how issues could be improved in the future’: Anthony Albanese', 'PM Anthony Albanese Signals Review Of Scott Morrison’s Conduct Is On The Horizon 23/08/2022', 'Standing ovation as Albo necks beer', 'Albo chugs beer at Gang of Youths concert', 'Aussies being left behind by the PM', 'Prime Minister’s big jobs promise', 'Huge push for foreign workers in Australia', 'Can we afford another mouth to feed?', 'SA projects under review after former PM’s power snatch', 'Albanese hints at ScoMo inquiry', '‘Never again’: Albo flags huge action']
https://www.couriermail.com.au/search-results?q=Anthony Albanese

Process finished with exit code 0

P.S. print and sleep statements are for easy visual. They are optional and could be omitted. If needed by explicit_wait like webdriverwait could be used.

CodePudding user response：

Your search should be of type WebElement but instead you define search as call of click method. Try to replace

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "/html/body/nav/form/button"))).click() 
search = driver.find_element(by=By.XPATH,value='/html/body/nav/form/button').click()
search.send_keys("Anthony Albanese")

with

search = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "/html/body/nav/form/button")))
search.click()
search.send_keys("Anthony Albanese")

CodePudding user response：

This code line

search = driver.find_element(by=By.XPATH,value='/html/body/nav/form/button').click()

Assigns nothing i.e. 'NoneType' object into the search object because

driver.find_element(by=By.XPATH,value='/html/body/nav/form/button')

returns a WebElement but it is not assigned to search object on the left side since you are applying a click() method on that returned WebElement while the click() method returns nothing.
You can fix this your code simply like the following:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "/html/body/nav/form/button"))).click() 
search = driver.find_element(by=By.XPATH,value='/html/body/nav/form/button')
search.click()
search.send_keys("Anthony Albanese")
time.sleep(10)
search.send_keys(Keys.ENTER)

You can also improve your code by removing the hardcoded delay of time.sleep(10) - Expected Conditions explicit waits will resolve most of these issues by significantly improving the text run time.
Also this XPath locator '/html/body/nav/form/button' can and should be improved.