I am trying to fill up a dataframe with the expected hourly irradiation for an specific location for the next 14 days. I am trying to do scrapping with Selenium from a website where those values are provided, but I do not know how to get the specific values and automate the overall process.

This is what I´ve tried: The df that I want to fill can be generated as follows:

days = range(1,15,1)
hours = ['00:00','01:00','02:00','03:00','04:00','05:00','06:00','07:00','08:00','09:00','10:00','11:00','12:00','13:00','14:00','15:00','16:00','17:00','18:00','19:00','20:00','21:00','22:00','23:00','24:00']
today = date.today()

dates = []
for d in days:
    dates.append(today   timedelta(days=d))

irradiation = pd.DataFrame(columns=dates, index=hours)
irradiation = irradiation.fillna(0) 

** Let me know if you find an easier way to create it.

With this dataframe, and by using Selenium, I want to field up the cells which values are provided in this website

So for this example, the result would look like:

We only fill up the hours where we have values, and they can change one day from other (so one day we have irradiation values from 07:00 and other from 09:00)

This is what I tried:

PATH = "C://Program Files (x86)//chromedriver.exe" # chromedriver.exe path
driver = webdriver.Chrome(PATH)

driver.get("https://www.tutiempo.net/radiacion-solar/madrid.html")
#close cookies
time.sleep(10)
for d in days:
   results = driver.find_element(By.ID, "HorasDia" str(d))
   # irradiation.loc[[h],[today   timedelta(days=(d-1))]] = 0 #here actual value of the irradiation the that hour
   print(results.text)

By doing this, I get the hour and the irradiation but I do not know how to iterate each hour and storage that into the df. Does anyone a solution for this?

CodePudding user response：

This is one way you can get that data as a dataframe:

import requests
import pandas as pd
from bs4 import BeautifulSoup as bs

url = 'https://www.tutiempo.net/radiacion-solar/madrid.html'

big_list = []
r = requests.get(url)
soup = bs(r.text, 'html.parser')
days = soup.select('h3')
for x in days:
    try:
        hours_div = x.find_next('div', {'class': 'horashidsow'})
        hours = hours_div.select('div.horhor')
        for hour in hours:
            h_div = hour.select_one('span.hora')
            val_div = hour.select_one('span.ener')
            big_list.append((x.text, h_div.text, val_div.text))
    except Exception as e:
        print('all done, stopping')
        break
df = pd.DataFrame(big_list, columns = ['Day', 'Hour', 'Value'])
print(df)

Result:

1

​

Relevant documentation for Requests: https://requests.readthedocs.io/en/latest/

For Pandas: https://pandas.pydata.org/pandas-docs/stable/index.html

And for BeautifulSoup: https://beautiful-soup-4.readthedocs.io/en/latest/index.html