How to automatically send a ``get request`` in a loop after some interval of time?-CodePudding

I am making a web application that is scrapping the news from a news site and saves to my database (scrapping is done just for learning purpose). After the database is updated all the stored data is sent to the user frontend.

Here is the route responsible for the above action. router.get('/news, postController.getNewsPost);

New news are added to the site being scrapped as the day passes. Lets say if no user logs in to my application my database does not updates because the route mentioned above does not fires.

I want my database to be updated periodically even when no users have logged into my web application.

I am new to backend development so please guide me on how i can achieve this, also let me know if more information is required.

CodePudding user response：

The simplest solution would be to use a setInterval to execute a specific function every N seconds:

setInterval(function() {
  // scrap news and save them to the database every 5 seconds
}, 5000)

More versatile and solid solutions can be implemented using an external library for scheduling task, something like Agenda

CodePudding user response：

You can use node-cron module to schedule your scrapping job.

Install node-cron using npm:

$ npm install --save node-cron

Import node-cron and schedule your scrapping job:

var cron = require('node-cron');

cron.schedule('* * * * *', () => {
  console.log('Scrapping....');
});

Here is the module link. https://www.npmjs.com/package/node-cron

CodePudding user response：

You can use the following method

$ npm install --save node-cron

var cron = require('node-cron');

cron.schedule('second minute hour day month', () => {
//update database code  


});

library link enter link description here