clicking a tab on a page to alow selenium to scrape-CodePudding

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from time import sleep
from datetime import datetime
import pandas as pd
import warnings
import os

warnings.filterwarnings('ignore')


url = 'https://www.fifa.com/fifaplus/en/match-centre/match/17/255711/285063/400128082?country=IE&wtw-filter=ALL'

option = Options()
option.headless = True
driver = webdriver.Chrome("C:/Users/paulc/Documents/Medium Football/chromedriver.exe",options=option)

# Scraping the data

HomeTeam = driver.find_element(By.XPATH, "/html/body/div[1]/main/div/div[1]/div/section/div[1]/div[1]/div[2]/div[3]/div/div[1]/div/div/div[1]/div[1]/p").text
AwayTeam = driver.find_element(By.XPATH, "/html/body/div[1]/main/div/div[1]/div/section/div[1]/div[1]/div[2]/div[3]/div/div[1]/div/div/div[3]/div[2]").text
Result = driver.find_element(By.XPATH, "/html/body/div[1]/main/div/div[1]/div/section/div[1]/div[1]/div[2]/div[3]/div/div[1]/div/div/div[2]").text

elem = WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div

Hi Guys, I am looking to scrape the world cup data, I have managed to start it easily by getting the team names and scores. The in game statistics are housed in the stats tab in the image. So to start scraping them I need to be able to click on it with selenium and make the page active.

Am i missing something obvious and that this cannot be done through an xpath?

help is appreciated.

CodePudding user response：

In case you want to click on "LINE UP" tab this can be done with the following XPath: "//div[text()='LINE UP']". To click on "STATS" tab you can use this XPath: "//div[text()='STATS']". So, Selenium code line can be as following:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()='LINE UP']"))).click()

For the LINE UP tab and

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()='STATS']"))).click()

for the "STATS" tab.
Also, you need to improve your locators. Very long absolute XPaths are very breakable.