How to click on a link while webscraping in python's selenium?-CodePudding

I am using selenium for python. When I am using the xpath to click on a link. I am getting an error TimeoutException: Message:.Ive tried using by.ID and by.tag but it seems like this link is hidden. How can I click these two links.

here my code for the first link:

btn = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH,"/html/body/div[3]/div/div/div[2]/div/div/div/div[2]/div/div/div[1]/div/div/div[2]/div/div/div")))
btn.click()


<div class="lib_33_IXqu lib_10OTPLG lib_rljvxxj lib_2bmVxh4 lib_AWe8PWK lib_NH5Lx3B lib_AWe8PWK"><div class="">Most Active<div class="lib_gdMpTuS lib_3Wb397t lib_QVji0M8 lib_1dwKEN3 lib_2IaUGOQ" aria-hidden="true">Most Active</div></div></div>

<div class="" data-selected="false"><div class="lib_33_IXqu lib_10OTPLG lib_rljvxxj lib_2bmVxh4 lib_AWe8PWK lib_NH5Lx3B lib_AWe8PWK"><div class="">Watchers<div class="lib_gdMpTuS lib_3Wb397t lib_QVji0M8 lib_1dwKEN3 lib_2IaUGOQ" aria-hidden="true">Watchers</div></div></div></div>

CodePudding user response：

This code should work. Additionally, I noticed that there's an overlay that pops up after a few seconds that can interrupt your mouse clicks. I've added a line of code to click out of it too.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time

# location of chromedriver.exe
driver = webdriver.Chrome("D:/chromedriver/94/chromedriver.exe")

driver.get("https://stocktwits.com/rankings/trending")

# waiting for the links to be available
WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '//div[@]')))

# capturing the links
links = driver.find_elements(By.XPATH, '//div[@]')

# get rid of the overlay message
WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '//button[@]'))).click()

# looping through the links since they all have the same class
for link in links:
    link.click()
    # do something
    time.sleep(2)

You could also access these links directly by their URLs:

stocktwits.com/rankings/most-active
stocktwits.com/rankings/watchers

I see that the overlay pops up a few times after maybe a minute. You could use a function to create a script:

def close_overlay():
    return """
setInterval(()=>{{var overlay = document.querySelector('button[]');
if(overlay){{overlay.click();}} }}, 5000);
"""

and later, call this in your script somewhere like this:

driver.execute_script(close_overlay())

This little script will check for the close button on that overlay every 5 seconds and closes it.

Note: This script could be trying to click on the close button at the same time your main bot is trying to click. This would lead to an ElementClickInterceptedException. You could handle this exception in your code.

This isn't required though, but something that might come in handy for you later.