selenium scraper get empty list with Xpath-CodePudding

I am scraping price data on the website: https://fbx.freightos.com/ .

This is the code below:

from selenium import webdriver from selenium.webdriver.edge.service import Service from selenium.webdriver.common.by import By

s = Service(e_driver_path) driver = webdriver.Edge(service=s)

elements = driver.find_elements(By.XPATH, /html/body/div[2]/div[1]/div[3]/div/div[1]/section/div[2]/div/div[2]/div/div[1]/div/div/div[1]/span)

content = "".join([element.text for element in elements])

print(content)

the problem is the result. It is an empty list. As planned, it should be the "Current FBX" and the result looks like "$3,540". Please help.

Thanks ahead.

CodePudding user response：

Try to wait for required data

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

fbx = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//div[.='Current FBX']/following-sibling::span[normalize-space()]"))).text

CodePudding user response：

Use webdriverwait() and wait for visibility of element located and following css selector

driver.get("https://fbx.freightos.com/")
print(WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.CSS_SELECTOR,".row.bottom-xs.between-xs div:nth-of-type(1)>div"))).text)
print(WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.CSS_SELECTOR,".row.bottom-xs.between-xs div:nth-of-type(1)>span"))).text)

You need to import below libraries.

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By

CodePudding user response：

This xpath will directly target the price web element. Try fetching this after some wait as mentioned by @KunduK

//div[contains(text(),'Current FBX')]/parent::div/span