I am using beautiful soup to try to get data from the Overwatch League Schedule website using beautiful soup, however, despite all the documentation saying that bs4 is capable of finding nested divs if i have their class it only returns an empty list.

here is the url: https://overwatchleague.com/en-us/schedule?stage=regular_season&week=1

here is what I am trying to get:

bs = BeautifulSoup(req.text, "html.parser")
matches = bs.find_all("div", class_="schedule-boardstyles__ContainerCards-j4x5cc-8 jcvNlt")

to eventually be able to loop through the divs in that and scrape the match data from it. However, it's not working and only returning a [], is there something I'm doing wrong?

CodePudding user response：

When a page is loaded in it often runs some scripts to fill in the information.

Beautifulsoup is only a parser and cannot render a page.

You will need something like selenium to render the page before using beautifulsoup to find the elements

CodePudding user response：

It isn't working since request is getting the html before the page is fully loaded. I don't think there is way to make it wait. You could try doing it with selenium