I'm new to Python and Selenium and having the following problem:

I want to scrape certain columns from a table. The table is a table of the German Bundesliga and there are two buttons above it to change the season and the matchday. I want to scrape the data only for the 2021/22 season, but for all matchdays.

I managed to get all rows of the table for one matchday, but I don't know how to "iterate" over all matchdays.

I greatly appreciate any help on this!

Below is my last try. The result list only gives me the first 5 rows per matchday. I don't understand why and I'm running out of ideas how to correct my code. I'm not sure if it's a good idea to use xpath here, but I couldn't figure out another way to find the correct column entries (maybe, it's better to use class instead of xpath).

What I would expect is a list containing all rows of the first table on the page for all 34 matchdays (18 x 34 rows).

element_list = []
  
for matchday in range(1, 35, 1):
    
    url = 'https://www.kicker.de/bundesliga/tabelle/2021-22/'   str(matchday)
    driver = webdriver.Chrome()
    driver.get(url)
    position = driver.find_elements(By.XPATH, "/html[@class=' kick__nodark']/body/div[@id='kick__page-container']/div[@id='kick__page']/div[@class='kick__area--main']/div[@class='kick__card '][1]/div[@class='kick__data-grid']/div[@class='kick__data-grid__main ']/div[@class='kick__site-padding']/div[@id='1']/div[@class='kick__module-margin']/table[@class='kick__table kick__table--ranking kick__table--alternate kick__table--resptabelle']/tbody/tr/td[@class='kick__table--ranking__rank kick__respt-m-o-1 kick__respt-m-w-25']")
    team = driver.find_elements(By.XPATH, "/html[@class=' kick__nodark']/body/div[@id='kick__page-container']/div[@id='kick__page']/div[@class='kick__area--main']/div[@class='kick__card '][1]/div[@class='kick__data-grid']/div[@class='kick__data-grid__main ']/div[@class='kick__site-padding']/div[@id='1']/div[@class='kick__module-margin']/table[@class='kick__table kick__table--ranking kick__table--alternate kick__table--resptabelle']/tbody/tr/td[@class='kick__table--ranking__teamname kick__table--ranking__index kick__t__a__l kick__respt-m-o-4 kick__respt-m-w-120 kick__t__a__l']/a/span[@class='kick__table--show-desktop']")
    matchday = driver.find_elements(By.XPATH, "/html[@class=' kick__nodark']/body/div[@id='kick__page-container']/div[@id='kick__page']/div[@class='kick__area--main']/div[@class='kick__card '][1]/div[@class='kick__data-grid']/div[@class='kick__data-grid__main ']/div[@class='kick__site-padding']/div[@id='1']/div[@class='kick__module-margin']/table[@class='kick__table kick__table--ranking kick__table--alternate kick__table--resptabelle']/tbody/tr/td[@class='kick__table--ranking__number'][1]")
    goals = driver.find_elements(By.XPATH, "/html[@class=' kick__nodark']/body/div[@id='kick__page-container']/div[@id='kick__page']/div[@class='kick__area--main']/div[@class='kick__card '][1]/div[@class='kick__data-grid']/div[@class='kick__data-grid__main ']/div[@class='kick__site-padding']/div[@id='1']/div[@class='kick__module-margin']/table[@class='kick__table kick__table--ranking kick__table--alternate kick__table--resptabelle']/tbody/tr/td[@class='kick__table--ranking__number'][2]")
    points = driver.find_elements(By.XPATH, "/html[@class=' kick__nodark']/body/div[@id='kick__page-container']/div[@id='kick__page']/div[@class='kick__area--main']/div[@class='kick__card '][1]/div[@class='kick__data-grid']/div[@class='kick__data-grid__main ']/div[@class='kick__site-padding']/div[@id='1']/div[@class='kick__module-margin']/table[@class='kick__table kick__table--ranking kick__table--alternate kick__table--resptabelle']/tbody/tr/td[@class='kick__table--ranking__master kick__respt-m-o-5']")
  
    for i in range(len(team)):
        element_list.append([position[i].text, team[i].text, matchday[i].text, goals[i].text, points[i].text])
  
# closing the driver
driver.close()

CodePudding user response：

Without selenium, I modified your code as follows for get the results you're looking for:

import requests
from bs4 import BeautifulSoup
import pandas as pd
import time as t

final_df = None # will contain the dataframe with all the data obtained

# Loop for make a request by each iteration - I've modified the "35" to "5" 
# for generate a shortened version of the desired results
for matchday in range(1, 5, 1):
  # Get the "table" HTML elements that exist on the page:  
  scrapper = pd.read_html('https://www.kicker.de/bundesliga/tabelle/2021-22/'   str(matchday))
  
  # In my tests, the 0 position is the one that contains the desired data:       
  for idx, table in enumerate(scrapper):
    if (idx == 0): 
      # Remove certain columns not needed: 
      table = table.drop(table.columns[[0, 1, 2]], axis=1)

      # Build the final dataframe - by initializing the variable 
      # or append the "table" to the final dataframe:
      if (final_df is None): 
        final_df = table
      else: 
        final_df = final_df.append(table, ignore_index=True)

# Display the final dataframe: 
display(final_df)

Results - shortened: