Im having trouble working through the following problem. Im trying to collect the data from the following webpage:https://localhelp.healthcare.gov/#/results?q=UTAH&lat=0&lng=0&city=&state=UT&zip_code=&mp=FFM
My method is to use the Selenium chrome driver to collect the data, for each healthcare agent, off this webpage, but dont know how I would loop through each record and add the data to each created list. So far, I can collect the data for a single record, but my problem is with my loop. How would I identify each record as an agent, and add that to my data frame for output? here is my code:
from selenium import webdriver # connect python with webbrowser-chrome
import time
import pandas as pd
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome('C:/Users/picka/Documents/chromedriver.exe')
driver.maximize_window()
url = 'https://localhelp.healthcare.gov/#/results?q=UTAH&lat=0&lng=0&city=&state=UT&zip_code=&mp=FFM'
name = []
phone = []
email = []
def go_to_network():
driver.get(url)
for agent in driver.find_elements_by_xpath('class.qa-flh-results-list'):
get_name = (WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "span.qa-flh-resource-name"))).text)
get_phone = (WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "a.qa-flh-resource-phone"))).text)
get_email = (WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "a.ds-u-overflow--hidden.ds-u-truncate.ds-u-display--inline-block"))).text)
name.append(get_name)
phone.append(get_phone)
email.append(get_email)
go_to_network()
record_output = {'Agent Name': name, 'Phone': phone, 'Email': email}
df = pd.DataFrame(record_output)
df.to_csv(r'C:\Users\picka\Documents\Dev\Reports\Agent-data.csv', header=True, index=False)
print(df)
CodePudding user response:
To extract and print all the Agent Name, Phone and Email using Selenium you can use List Comprehension inducing WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:
Code Block:
driver.get('https://localhelp.healthcare.gov/#/results?q=UTAH&lat=0&lng=0&city=&state=UT&zip_code=&mp=FFM') get_name = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "span.qa-flh-resource-name")))] get_phone = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.qa-flh-resource-phone")))] get_email = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.ds-u-overflow--hidden.ds-u-truncate.ds-u-display--inline-block")))] for i,j,k in zip(get_name, get_phone, get_email): print(f"{i}'s' phone number is {j} and email is {k}") driver.quit()
Console Output:
Wesley Elton's' phone number is (801) 404 - 2424 and email is [email protected] Raquel Bell's' phone number is (801) 842 - 2870 and email is [email protected] Brandon Berglund's' phone number is (801) 981 - 9414 and email is [email protected] Steven Cochran's' phone number is (801) 800 - 8360 and email is [email protected] victoria dang's' phone number is (801) 462 - 5190 and email is [email protected] Dan Jessop's' phone number is (435) 232 - 8833 and email is [email protected] Billy Gerdts's' phone number is (801) 280 - 1162 and email is [email protected] Michael Saldana's' phone number is (801) 879 - 1032 and email is [email protected] Brandon Johnson's' phone number is (435) 249 - 0725 and email is [email protected] Matthew Selph's' phone number is (801) 918 - 3945 and email is [email protected]
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC