I want to scrape a website every minute from now until the end of time. It would roughly be one megabyte in size per scrape, and I wish to scrape it and put the relational data in a SQL managed instance on Azure.

My question is does one use Azure functions or Azure VM to do this? I also would like to use dash (plotly) to have a web app that posts real time updated information, but I don't mean to get ahead of myself.

Thanks for the advice!

CodePudding user response：

• What you can do is use a time trigger function. This function will get activate every day at a specific developer defined time and it will execute whatever you want, and it will execute at a specified time.

• In time trigger there is concept of time NCrontab which dictates the time at which the time trigger will gets trigger. Since you want to scrape the website every minute use the following Ncrontab expression 0 */1 * * * *.

• Now to connect to the sql database you can do this by either using managed identities and connection string. Using connectionstring is easier just Store the connection string in the app setting and red then as if they are enviorment variables.

• Now all you have to do is make http request to the webisite and store them in sql.

Reference:

Azure time trigger

Connection to database

Make Http request in c#