Python Scrape Weather Site Returns Nothing-CodePudding

I'm trying to scrape a forecast table from wunderground. When inspecting the website, I see this:

Here is the code...When I run this, I get no results.

  import requests
  from bs4 import BeautifulSoup


   URL = "https://www.wunderground.com/calendar/us/ca/santa-barbara/KSBA"
   page = requests.get(URL)

   soup = BeautifulSoup(page.content, "html.parser")

   for ultag in soup.find_all('ul', {'class': 'calendar-days'}):

     for litag in ultag.find_all('li'):
        print (litag.text)

any help is appreciated.

CodePudding user response：

As the site is using Javascript to generate the weather data, you will need to use something like Selenium to extract the required info. Using a combination of bs4 and Selenium - and with an appropriate sleep time, the weather data can be extracted.

Here is the code:

Code:

from bs4 import BeautifulSoup
from selenium import webdriver
from time import sleep

# Open browser and navigate to page
driver = webdriver.Chrome()
driver.get("https://www.wunderground.com/calendar/us/ca/santa-barbara/KSBA")

# Adjust sleep time to suit your needs
sleep(10)

html_source = driver.page_source
soup = BeautifulSoup(html_source, "html.parser")

# Find the weather data
for ultag in soup.find_all('ul', {'class': 'calendar-days'}):
    for litag in ultag.find_all('li'):
        print(litag.text)

driver.close()

Output:

27mostly sunnyActual:65° | 39°0 °in
28partly cloudyActual:62° | 48°0 °in
29mostly sunnyActual:61° | 40°0 °in
30cloudyActual:62° | 42°0 °in
1cloudyActual:59° | 53°0 °in
2cloudyActual:59° | 54°0 °in
3cloudyActual:64° | 57°0.56 °in
4cloudyActual:61° | 48°0 °in
5mostly sunnyActual:65° | 43°0.45 °in
6partly cloudyActual:61° | 41°0 °in
7partly cloudyActual:60° | 39°0 °in
8mostly sunnyActual:59° | 41°0 °in
9mostly sunnyActual:63° | 39°0 °in
10scattered showersActual:58° | 46°0 °in
11scattered showersActual:59° | 40°0.86 °in
12cloudyActual:56° | 42°0.25 °in
13mostly sunnyActual:58° | 33°0.14 °in
14mostly sunnyActual:59° | 37°0 °in
15mostly sunnyActual:58° | 42°0 °in
16mostly sunnyActual:65° | 41°0 °in
17mostly sunnyActual:62° | 37°0 °in
18mostly sunnyActual:61° | 35°0 °in
19mostly sunnyActual:61° | 38°0 °in
20mostly sunnyActual:62° | 40°0 °in
21mostly sunnyActual:64° | 41°0 °in
22mostly sunnyActual:65° | 41°0 °in
23mostly sunnyActual:64° | 42°0 °in
24mostly sunnyActual:74° | 41°0 °in
25mostly sunnyActual:78° | 43°0 °in
26mostly sunnyActual:72° | 44°0 °in
27scattered showersActual:60° | 53°0 °in
28mostly sunnyActual:62° | 49°1.01 °in
29RainForecast:60° | 50°0.1 °in
30Mostly CloudyForecast:62° | 51°0.05 °in