Python Selenium Chrome - send_keys() not sending keys during 2nd iteration of loop when scraping Whi-CodePudding

The

CodePudding user response：

One of the scripts is reopening the page if any input is not empty when the script runs.
(There are too many scripts on the page; I was not able to narrow down which one it is.)

A simple solution is to add sleep(1) after driver.get(...).

driver.get('https://www.whitepages.com/') sleep(1) # Add this

Power usage

sleep(1) significantly slows down the script, especially if looping through many names.

Instead, we can use two tabs alternately and prepare each tab for the next-next iteration.

Setup before loop:

driver.get('https://www.whitepages.com/') driver.execute_script('window.open("https://www.whitepages.com/")') driver.switch_to.window(driver.window_handles[0]) next_index = 1

Prepare for the next and next-next iterations after processing each name:

for name in names: ... # Prepare this tab for next-next iteration driver.get('https://www.whitepages.com/') # Switch to another tab for next iteration driver.switch_to.window(driver.window_handles[next_index]) next_index = 1 - next_index

Comment out driver.get(...) in scrape_individual_info_wp function:

# driver.get('https://www.whitepages.com/')

CodePudding user response：

Try with below code once.

Navigate to the Website once and can use driver.back() to come back to the original Page.

User Explicit wait to wait for the elements to appear. And can use good locators like ID or CLASS_NAME to locate the elements.

from selenium.webdriver.support.wait import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium import webdriver from selenium.webdriver.chrome.service import Service from selenium.webdriver.common.by import By import time def scrape_individual_info_wp(driver, individual_name, city_state): name_field = wait.until(EC.element_to_be_clickable((By.ID, "desktopSearchBar"))) name_field.clear() # Clear the field to enter a new name name_field.send_keys(individual_name) state_field = wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "pa-3"))) state_field.clear() # Clear the field and enter the state state_field.send_keys(city_state) search = wait.until(EC.element_to_be_clickable((By.ID, "wp-search"))) search.click() time.sleep(2) # Using Sleep to make sure the data is loaded. Can use Waits to wait for the data to appear and extract the same. # Code to scarpe data. driver.back() driver = webdriver.Chrome(service= Service("C:/expediaproject/Chromedriver/chromedriver.exe")) driver.maximize_window() wait = WebDriverWait(driver,30) driver.get("https://www.whitepages.com/") names = ['Kevin J Haggerty', 'Patricia B Halliday', 'David R Harb', 'Jeffrey E Hathway', 'Hanshin Hsieh'] state = "Manchester, NH" for name in names: scrape_individual_info_wp(driver, name, state) driver.quit()