I made this little script as I'm learning to scrape with bs4 and selenium and I want to collect periodic date of weather into a csv file but I am quite new and I don't know where to begin.

Can someone point me in the right direction on to how to make this run every date and append data? For now, I run and it just makes the header and the second row.

from bs4 import BeautifulSoup from selenium import webdriver from selenium.webdriver.common.by import By from csv import writer

url = 'https://weather.com/es-US/tiempo/10dias/l/468124beb45cd6817ec6a1da2675f104fede76a75ba7ce08c9ae21606ac208b5'
driver = webdriver.Chrome('/Users/thras/Desktop/Chromedriver/chromedriver')
driver.get(url)
source = driver.page_source

soup = BeautifulSoup(source, 'html.parser')



degrees = soup.find('span', class_="DailyContent--temp--3d4dn").text
humidity = soup.find('span', class_="Wind--windWrapper--3aqXJ DailyContent--value--37sk2").text
wind = soup.find('span', class_="DailyContent--temp--3d4dn").text
date = soup.find('span', class_="DailyContent--daypartDate--2A3Wi").text

info = [degrees, humidity, wind, date]

with open('weather.csv', 'w', encoding='utf8', newline='') as f:
    mywriter = writer(f)
    header = ('Degrees', 'Humidity', 'Wind', 'Date')
    mywriter.writerow(header)
    mywriter.writerow([degrees, humidity, wind, date])
    
driver.quit()

CodePudding user response：

To schedule the python script to run it every day you could either use a builtin task scheduler of your os or you could schedule it inside the script itself. Here you could use the python library schedule. You can see an example here: stackoverflow schedule

To add the row to your existing csv, you just have to open it in append mode.

CodePudding user response：

I think all you need to do is to schedual your programm to be executed every day. If you are on MacOs or Linux you can use crontab.

If you are on Windows you can use Windows Task Scheduler search on the interent how they work.