TimeoutException due to a change of tags?-CodePudding

I am having some problems with TimeoutException.

My code has always run fine in the past days but today something has changed since I am getting the TimeoutException error message. More likely is due to a locator change on the website.

My code is structured as follows:

# Initialise Chrome parameters
chrome_options = webdriver.ChromeOptions()

# Open Chrome
driver=webdriver.Chrome(path,chrome_options=chrome_options)
driver.maximize_window()
    
response=driver.get(my_web)


wait = WebDriverWait(driver, 30)
time.sleep(randrange(5))
driver.execute_script("window.scrollTo(0, 1000)")


message = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,"section.section div.container h2"))).text
        t_score = wait.until(EC.visibility_of_element_located((By.XPATH, "//div[text()='Trustscore']/../following-sibling::div/descendant::div[@class='icon']"))).text

I have tried to change the time wait even to 60sec, but nothing has changed since I a getting the same error.

The error is returned due to the following: section.section div.container h2. I am interested in Trustscore.

URL here

CodePudding user response：

Your paths are incorrect. I assume the html has changed. You could use:

message = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,".tile p > strong"))).text
t_score = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".trustscore-rating"))).text

However,

You need a better wait condition as the rating can be slow to fully update leading to printing an incorrect value.
There are anti-bot cloudflare measures meaning scraping is likely against T&C and you will get a captcha at some point and possibly a ban.

CodePudding user response：

Not sure what HTML the website had earlier, But I do this outer HTML for Trustscore.

<div class="trustscore-rating"><span style="color: rgb(0, 177, 106);">100</span> / 100</div>

I have looked upon the HTMLDOM and we have a a unique entry for this div.

Please use this css_selector :

div.trustscore-rating

Code 1 :

message = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.trustscore-rating"))).text

message = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.trustscore-rating"))).get_attribute('innerText')