I have been recently challenged with an architectural problem. Basically, I developed a Node.js application that fetches three zip files from Census.gov (13 MB, 1.2 MB and 6.7 GB) which takes about 15 to 20 minutes. After the files are downloaded the application unzips these files and extracts needed data to an AWS RDS Database. The issue for me is that this application needs to run only one time each year. What would be the best solution for this kind of task? Also, the zip files are deleted after the processing is done.
CodePudding user response：
I would use a cron job. You can use this website (https://crontab.guru/every-year) to determine the correct settings for the crontab.
0 0 1 12 1
This setting will run “At 00:00 on day-of-month 1 and on Monday in December.”
To run the nodeJS program you simply put node yourcode.js aftewards. So it would look like the code below. Where node is you may need to put the path to the node program, and where yourprogram.js is you simply need to add the path there as well.
0 0 1 12 1 node yourprogram.js
CodePudding user response：
Hei, I would give u suggestion. But according what Services do you use. In example if using Google Cloud with Google Scheduller. If using Openshift or another u can use Cronjob. But it worst case configuration I think where u need make some yaml file deployment that need trigger to publisher/subscriber:
- Make some subscriber, on services which can trigger by Google PubSub by Topic to do your task and after all executed publish to the broker (Google PubSuB) again.
- And than make another subscriber to trigger deleting file after receive a publisher message if all task execute.
The Idea i suggest because the process like that, it best practices if using the Asyncrhrounus process.
Thanks,