I am trying to Web scrape Indeed Jobs using both selenium and beautiful soup, I am able to extract all the details from the Job but to get the Job description I have used Selenium, But when I am trying to find the Job-description Id it's returning the error: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="jobDecriptionText"]"} (Session info: chrome=100.0.4896.75)

I have used the following code:

for jobs in analyst_jobs:
get_html = jobs.get_attribute('innerHTML')
soup = BeautifulSoup(get_html, 'html.parser')

title = soup.find('h2', {'class': 'jobTitle'}).text
print(title)
company = soup.find('span', {'class': 'companyName'}).text
print(company)
location = soup.find('div', {'class': 'companyLocation'}).text
print(location)
desc = soup.find('div', {'class': 'job-snippet'}).text
print(desc)
summary = jobs.find_elements(By.CLASS_NAME, "job_seen_beacon")[0]
summary.click()
jd = driver.find_element(By.ID, "jobDecriptionText")
print(jd)

CodePudding user response：

As the error says: Unable to locate element. Selenium may work faster before the element is located. You can use "wait" to wait until your element is located. For more, you can see: https://selenium-python.readthedocs.io/waits.html

So you can do that as follows:

#import necessary parts for wait
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

#..your code for webdriver

#set wait to make it easy to use
wait = WebDriverWait(driver, 10)
 
#your other codes... 

#finally, you can use wait like this:
jd = wait.until(EC.presence_of_element_located((By.ID, 
"jobDecriptionText")))
print(jd)

CodePudding user response：

There are two things happening when no such element exception is thrown

1. wrong locator - please check if you are using correct id/xpath.
2. WebElement is not yet loaded - you need to add wait for the element to be interactable.

CodePudding user response：

Just adding in the Furkan answer, you can use the below one as well.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

#..your code for webdriver

#set wait to make it easy to use
wait = WebDriverWait(driver, 10)
 
#your other codes... 

#finally, you can use wait like this:
jd = wait.until(EC.visibility_of_element_located((By.ID, 
"jobDecriptionText")))
print(jd)