I need reflection, vision and documentation on my problem.

I wrote a python script to calculate something from an API and export the result in a CSV file. Then, I use a JavaScript script to display the data from this CSV file on a building website. I need to have the latest data available for my website, so I opened a VM instance in Google Cloud Platform (Google Compute Engine) and set a Crontab job to run automatically my python script. The calculation is now executed every day and the result is exported to the CSV file, but stored in this VM instance.

Here is my goal: How can I get my CSV file on my website? The CSV is always on the virtual machine and I do not know how to communicate with my JavaScript script to the VM. Do I have to communicate directly with the VM? Do I have to go through another step before (server, API, etc.)?

I cannot find a specific solution for my problem on the internet.

Thanks in advance.

CodePudding user response：

I would suggest you a different approach than using a VM for calculating and storing your CSV.

The idea would be using a Python Cloud Function instead, that will be run by Cloud Scheduler and Pub/Sub.

This function will generate your CSV file, that will be stored on Cloud Storage. Here you can find an example of how to upload an object to Cloud Storage using Python.

Then, you need to give your website the ability to access that CSV file in Cloud Storage when required. As indicated by @guillaumeblaquiere, the exact way will be dependent on where your website is hosted: especially, it will constraint the auth mechanisms that you need to use to download your data. The documentation in GCP provides several examples about this matter.

The documentation of Google Cloud provides an example of the proposed architecture.

I came across an article that, in a certain way, describes the setup I suggested as well.

Please, consider review the costs of the different mentioned products.

CodePudding user response：

How can I get my CSV file on my website?

By making your python script output the CSV into your website's root folder.

Example, if you're running apache, chances are your root folder is somewhere in /var/www/html/...

If the script is generated from another machine (not the one with your website), then I would host it and make the server hosting your website fetch it via cronjob.

Basically: If your CSV is generated from the same machine as the website that will use it - simply output it to the website's folder

If your CSV is generated from another machine, make it publically accessible and have your website's machine cronjob fetch that CSV a few minute after it's generated.