Python Selenium XPATH not pulling any data-CodePudding

I have been trying to run this python code but nothing comes up when I print the variable table. Here's my code:

from selenium import webdriver
from selenium.webdriver.common.by import By
import pandas as pd
from bs4 import BeautifulSoup
import time

browser = webdriver.Firefox()

url = r'https://careers-techcu.icims.com/jobs/search?ss=1&mobile=false&width=980&height=500&bga=' \
  r'true&needsRedirect=false&jan1offset=-480&jun1offset=-420'

browser.get(url)

src = browser.page_source
parser = BeautifulSoup(src, 'lxml')
time.sleep(3)
titles = browser.find_elements(By.XPATH, '/html/body/div[2]/div[7]/div[1]/div[3]/a/h2')


print(titles)

The output is:

[]

CodePudding user response：

That info you're looking for is in an iframe. This is one way of getting that information:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument('disable-notifications')
chrome_options.add_argument("window-size=1280,720")

webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)

url = 'https://careers-techcu.icims.com/jobs/search?ss=1&mobile=false&width=980&height=500&bga=true&needsRedirect=false&jan1offset=-480&jun1offset=-420'
browser.get(url) 
WebDriverWait(browser, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//*[@id='icims_content_iframe']")))
print('switched to iframe')
jobs = WebDriverWait(browser,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div[class='col-xs-12 title']")))
for job in jobs:
    print(job.find_element(By.TAG_NAME, 'h2').text)

This will print in terminal:

switched to iframe
Member Services Specialist I
Senior Relationship Banker
Head of Engineering
Senior Consumer Loan Funder
Relationship Banker
Senior Relationship Banker
AVP, Market Development Officer
Universal Banker
Retail Banking Development Specialist
Mortgage Servicing Systems and Compliance Analyst
Senior Relationship Banker
AVP, Retail Bank Universal Support Manager
Senior Network Administrator
Relationship Banker
Strategic Lending Credit Products Manager
Relationship Banker

Of course you can select other info as well (job url, description, etc). Selenium docs can be found at https://www.selenium.dev/documentation/