I have been building a webscrapper bot with Python and Selenium, but have run into an issue. The website I am using to scrape information has a fieldset html tag with 4 label tags. I am specifically looking to click one of these label tags, but all of them have the same class name. I was hoping to see if anyone had some insight on how I can work around this issue. I have tried out a couple of things but unfortunately none are working.
This is the HTML code:
<fieldset>
<legend >Are you a?</legend>
<input type="hidden" name="_token" value="Nw3CU75ZNiiwSvcXofVUm3hOpcFkvzEc8UdjlPcX">
<span id="errMsg"></span>
<div aria-live="assertive" ></div>
<label tabindex="0" for="radio1">
<input tabindex="-1" id="radio1" name="radioStep1" type="radio" value='in-network'
aria-describedby="in-network-balloon">
<span style="vertical-align: top;">
<i ></i>
<i style="color:#fff" ></i>
</span>
<span id='in-network-balloon'>
Staying In-Network <span class='sr-only'>If your provider takes your insurance</span>
<i data-toggle="tooltip"
data-html="true" data-trigger="focus keypress click" aria-hidden="true"
title="<h3>Staying In-Network</h3>
<p>If your provider takes your insurance</p>"></i>
</span>
<mark >
<span >Next
<i ></i>
</span>
</mark>
</label>
<label tabindex="0" for="radio2">
<input tabindex="-1" id="radio2" name="radioStep1" type="radio" value='out-network'
aria-describedby="out-network-balloon">
<span style="vertical-align: top;">
<i ></i>
<i style="color:#fff" ></i>
</span>
<span id='out-network-balloon'>
Going Out-of-Network <span class='sr-only'>If your provider does not take your insurance.</span>
<i data-toggle="tooltip"
data-html="true" data-trigger="focus keypress click" aria-hidden="true"
title="<h3>Going Out-of-Network</h3>
<p>If your provider does not take your insurance.</p>"></i>
</span>
<mark >
<span >Next
<i ></i>
</span>
</mark>
</label>
<label tabindex="0" for="radio3">
<input checked tabindex="-1" id="radio3" name="radioStep1" type="radio" value='not-sure'
aria-describedby="not-sure-balloon">
<span style="vertical-align: top;">
<i ></i>
<i style="color:#fff" ></i>
</span>
<span id="not-sure-balloon">
Not Sure <span class='sr-only'>If you don't know if your provider takes your insurance</span>
<i data-toggle="tooltip"
data-html="true" data-trigger="focus keypress click" aria-hidden="true"
title="<h3>Not Sure</h3>
<p>If you don't know if your provider takes your insurance</p>"></i>
</span>
<mark >
<span >
Next
<i ></i>
</span>
</mark>
</label>
<label tabindex="0" for="radio4">
<input tabindex="-1" id="radio4" name="radioStep1" type="radio" value='uninsured'
aria-describedby="uninsured-balloon">
<span style="vertical-align: top;">
<i ></i>
<i style="color:#fff" ></i>
</span>
<span id="uninsured-balloon">
Uninsured <span >If you don't have health insurance or if your insurance pays for providers who aren’t in your plan.</span>
<i data-toggle="tooltip"
data-html="true" data-trigger="focus keypress click" aria-hidden="true"
title="<h3>Uninsured</h3>
<p>If you don't have health insurance or if your insurance pays for providers who aren’t in your plan.</p>"></i>
</span>
<mark >
<span >
Next
<i ></i>
</span>
</mark>
</label>
</fieldset>
Here is an image of what the page looks like. I am specifically looking to press the "Not Sure" section, and then "next"
I have tried this code
label_elements = driver.find_elements_by_css_selector("custom-control custom-radio")
label_elements[1].click()
along with this
label_elements = driver.find_elements_by_css_selector("custom-control custom-radio")
for label_element in label_elements:
label_element.click()
and a couple different variations of the above code, but it still has not been working.
Here is my full code:
import selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
import time
import os
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
s=Service('/Users/[name]/Desktop/chromedriver')
chromeOptions = Options()
chromeOptions.headless = False
driver = webdriver.Chrome(service=s, options=chromeOptions)
list_data = []
def initalize_browser():
driver.get("https://www.fairhealthconsumer.org/")
driver.maximize_window()
driver.find_element(By.XPATH, '//button[@]').click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@id='radio1' and normalize-space()='Einloggen']"))).click()
driver.find_element(By.ID, 'not-sure-balloon').click()
driver.find_elements(By.CLASS_NAME, 'fa-chevron-square-right')[2].click()
print(driver.page_source)
initalize_browser()
driver.quit
Here is the link: fairhealthconsumer.org/medical
Any guidance is super appreciated, thank you!
CodePudding user response：
Try this:
list_data = []
def initalize_browser():
driver.get("https://www.fairhealthconsumer.org/")
driver.maximize_window()
driver.find_element(By.XPATH, '//button[@]').click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, ".//*[@for='radio3']/span"))).click()
driver.find_element(By.XPATH, ".//*[@for='radio3']//span[@class='pt-2 border-top border-white']").click()
print(driver.page_source)
initalize_browser()
CodePudding user response：
why not use this for the 'not sure' button:
driver.find_element_by_id('not-sure-balloon').click()
and then this for the 'next' button
driver.find_elements_by_class_name('fa-chevron-square-right')[2].click()
Though this is usually the recommended way:
from selenium.webdriver.common.by import By
...
driver.find_element(By.ID, 'not-sure-balloon').click()
driver.find_elements(By.CLASS_NAME, 'fa-chevron-square-right')[2].click()
CodePudding user response：
The id of the not sure button is: not-sure-balloon
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
s = Service("D:\chromedriver_win32\chromedriver.exe")
browser = webdriver.Chrome(service=s)
url = "https://fairhealthconsumer.org/medical"
browser.get(url)
browser.find_element(By.ID, 'not-sure-balloon').click()