everyone, i'm doing a Scrap course, basically im stracting data from a web using webdriver and selenium. Everything work as intended until yesterday, today i can't print the info at all, i tried looking for errors or some advice, but im to noob to understand some tips, and i cant find any error in the code, as i said before, everything worked fine til yesterday.
here's the code:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
website = "https://www.adamchoi.co.uk/teamgoals/detailed"
path = r"H:/PROGRAMMING/Learning and Proyects/chromedriver_win32/chromedriver"
driver = webdriver.Chrome(executable_path= f"{path}")
driver.get(website)
all_matches_button = driver.find_element_by_xpath('//label[@analytics-event="All matches"]')
all_matches_button.click()
dropdown = Select(driver.find_element_by_id("country"))
dropdown.select_by_visible_text("Spain")
matches = driver.find_elements_by_tag_name("tr")
for match in matches:
match.txt
print(match.txt)
At the end i receive this: "H:\PROGRAMMING\anaconda\envs\Learning and Proyects\python.exe" "H:/PROGRAMMING/Learning and Proyects/Test scrape yelp.py" Process finished with exit code 0
and it is supposed to print the matches o,o
Sorry for the long post, i have no idea about how to fix it o,o
Thanks in advance :)
CodePudding user response:
I can't see how this code could work ever. The only way this could work is to run it in debug mode step by step, but not in a normal run mode.
You are missing waits here.
These should preferably be Expected Conditions explicit waits.
Please try the following:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select
import time
website = "https://www.adamchoi.co.uk/teamgoals/detailed"
path = r"H:/PROGRAMMING/Learning and Proyects/chromedriver_win32/chromedriver"
driver = webdriver.Chrome(executable_path= f"{path}")
wait = WebDriverWait(driver, 20)
driver.get(website)
wait.until(EC.visibility_of_element_located((By.XPATH, '//label[@analytics-event="All matches"]'))).click()
wait.until(EC.visibility_of_element_located((By.ID, 'country')))
dropdown = Select(driver.find_element_by_id("country"))
dropdown.select_by_visible_text("Spain")
wait.until(EC.visibility_of_element_located((By.XPATH, '//tr')))
time.sleep(0.5)
matches = driver.find_elements_by_tag_name("tr")
for match in matches:
print(match.text)