Python Selenium driver.find_element().text returns empty string, but text is visible in the driver.p-CodePudding

I'm trying to scrape some titles of the videos and to do so I'm using Selenium, but I've encountered a problem. driver.find_element().text returns empty string, but title is for sure located in given XPATH. Here is the fragment of the page source returned by driver.page_source:

<div ><a href="/f/4n3x7e31hpwxm8"target="_blank">Big.Sky.S03E01.ITA.WEBDL.1080p</a></div>

To find the title I am trying to use:

title_from_url = driver.find_element(
    By.XPATH, '//div[contains(@class, "title")]/a'
    ).text

From what I've read it could be caused by the fact that the page might not be fully loaded (I wasn't using any wait condition here). After that I've tried to add a wait condition and even time.sleep(), but it didn't change anything. <mini question: how would proper wait staitment look like here?>

Any help will be appreciated. Best regards, Ed.

Example site: https://mixdrop.to/e/4n3x7e31hpwxm8.

CodePudding user response：

You have to wait for element to be completely loaded before extracting it text content. WebDriverWait expected_conditions explicit waits should be used for that.
This should wait in case the element is visible on the page and the locator is correct:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 20)

title_from_url = wait.until(EC.visibility_of_element_located((By.XPATH, '//div[contains(@class, "title")]/a'))).text

CodePudding user response：

If you suspect its because of sync issue. You can use selenium waits.Let it be implicit of explicit.

Implicit: objdriver.implicitely_wait(float)

Explicit:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

objwait=WebDriverWait(driver,float,poll_frequency=float,ignored_exception=float)

objelement=objwait.until(EC.visibility_of_element_located((By.XPATH,"Your XPATH")))