Google search with Selenium Python doesn't work-CodePudding

I would like to retrieve the phone number of some points of interest from the Google results page with Selenium Python. I can accept the terms of use of Google but I can neither enter the query nor press the button. This is my code:

rom selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

url = 'https://www.google.it/'

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 20)
driver.get(url)

time.sleep(0.5)

# Accept the Google terms of use
driver.execute_script('return document.querySelector("#L2AGLb > div")').click()

names = ['Ristorante Roma Antica Roma', 'Ristorante e Braceria Al Piave Roma']

for name in names:
    input_search = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'gLFyf gsfi')))
    input_search.clear()
    input_search.send_keys(name)

    time.sleep(0.5)

    wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'gNO89b'))).click()

    time.sleep(0.5)

    # Extract the phone number here
    try:
        phone_number = ...
    except:
        pass

    driver.execute_script("window.history.go(-1)")

I should extract the phone number from the panel on the right of the page after clicking the button "Search" (if the panel is available).

Thanks in advance for your suggestions.

ADDENDUM

To find the phone numbers: inspecting the pages I have to search within the following span tags ('LrzXr zdqRlf kno-fv' is unique):

<span ><a data-dtype="d3ph" data-local-attribute="d3ph" jscontroller="LWZElb" href="#" jsdata="QKGTRc;_;B2Cx5o" jsaction="rcuQ6b:npT2md;F75qrd" data-ved="2ahUKEwik7rXAlNr2AhUFgP0HHedMA-IQkAgoAHoECDcQAw"><span><span aria-label="Chiama il numero di telefono 06 7047 6283">06 7047 6283</span></span></a></span>

and

<span ><a data-dtype="d3ph" data-local-attribute="d3ph" jscontroller="LWZElb" href="#" jsdata="QKGTRc;_;B4wf3Y" jsaction="rcuQ6b:npT2md;F75qrd" data-ved="2ahUKEwiwzs3blNr2AhU477sIHe5TA3sQkAgoAHoECEgQAw"><span><span aria-label="Chiama il numero di telefono 06 484467">06 484467</span></span></a></span>

My goal is to retrieve '06 7047 6283' and '06 484467'.

I succeeded using regular expressions but I would like to avoid them if possible:

content = wait.until(EC.visibility_of_element_located((By.XPATH, '//*[@id="kp-wp-tab-overview"]'))).text
phone_number = re.findall(r'Telefono: ([0-9 ] )', content)[0]

CodePudding user response：

I'm not sure what is this line here for:

driver.execute_script('return document.querySelector("#L2AGLb > div")').click()

I can't see any element matching that locator there and it throws an exception for me.
Also I have fixed the locator of search input field and the way to click Enter there so my code is:

names = ['Ristorante Roma Antica', 'Ristorante e Braceria Al Piave']

for name in names:
    input_search = wait.until(EC.visibility_of_element_located((By.XPATH, "//input[@name='q']")))
    input_search.clear()
    input_search.send_keys(name   Keys.ENTER)

    phone = wait.until(EC.visibility_of_element_located((By.XPATH, "//span[contains(@aria-label,'Call phone number')]"))).text
    print(phone)
    driver.execute_script("window.history.go(-1)")

    time.sleep(0.5)

And it works.
At lest it opens the search results.

CodePudding user response：

There are few things in your code that can be improved:

for input use google input name q which has been there from last 5-6 years or may be even more.
You can directly send input, you don't need to explicitly have #wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'gNO89b'))).click()

Use the below XPath:

//a[text()='Phone']/../following-sibling::span/descendant::span[@aria-label]

to extract the phone numbers, based on Phone text.

Code:

names = ['Ristorante Roma Antica Roma', 'Ristorante e Braceria Al Piave Roma']

for name in names:
    input_search = wait.until(EC.visibility_of_element_located((By.NAME, 'q')))
    input_search.clear()
    input_search.send_keys(name, Keys.RETURN)

    time.sleep(0.5)

    #wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'gNO89b'))).click()

    #time.sleep(0.5)

    # Extract the phone number here
    try:
        phone_number = wait.until(EC.visibility_of_element_located((By.XPATH, "//a[text()='Phone']/../following-sibling::span/descendant::span[@aria-label]")))
        print(phone_number.text)
    except:
        pass

    driver.execute_script("window.history.go(-1)")

Imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Output:

 39 06 7047 6283
 39 06 484467

Process finished with exit code 0