selenium BeautifulSoup find_all don't find all :)?-CodePudding

I'm trying to do an automation for matches scores. I'm using selenium and BeautifulSoup. The script get me only the first 10 matches and not others. How can I fix that?

from bs4 import BeautifulSoup
import lxml
from webdriver_manager.chrome import ChromeDriverManager
import time
import os
team_both = []
team_one = []
team_two = []

driver = webdriver.Chrome(ChromeDriverManager().install())
time.sleep(1)
os.system('clear')
time.sleep(1)
driver.get('https://www.bet365.de/#/IP/B1')

time.sleep(1)

driver.find_element_by_class_name('iip-IntroductoryPopup_Cross').click()
time.sleep(1)
driver.get('https://www.bet365.de/#/IP/B1')
time.sleep(1)

soup = BeautifulSoup(driver.page_source, 'lxml')

time.sleep(1)

tab_matches = soup.find_all('div', {"class" : "ovm-Fixture_Container"})

print(len(tab_matches))

CodePudding user response：

The problem is that you don't scroll the webpage, so all elements don't load. I added the functionality to scroll the page to the bottom and it works for me. I also removed BeautifulSoup, because it makes no sense to use it when you are already using Selenium.

from webdriver_manager.chrome import ChromeDriverManager
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains

team_both = []
team_one = []
team_two = []

driver = webdriver.Chrome(ChromeDriverManager().install())
time.sleep(1)
#os.system('clear')
time.sleep(1)
driver.get('https://www.bet365.de/#/IP/B1')

time.sleep(1)

driver.find_element_by_class_name('iip-IntroductoryPopup_Cross').click()
time.sleep(1)
driver.get('https://www.bet365.de/#/IP/B1')
time.sleep(1)

# Scroll to bottom
element = driver.find_element_by_class_name("ovm-OverviewScroller-enabled")
actions = ActionChains(driver)
actions.move_to_element(element).click_and_hold().perform()

time.sleep(5)

tab_matches = driver.find_elements(by=By.CLASS_NAME, value="ovm-Fixture_Container")

print(len(tab_matches))