We wanted to run (poll) scala/python code in GCP VM continuously which will run ETL program only when there is change in value in BQ table. i.e. we'll add which ETL to run in BQ table and based on that ETL program will run via GCP VM.

If there are multiple values in BQ table, it should run multiple ETL program simultaneously.

How we can achieve the same.

CodePudding user response：

For running your piece of code whenever BQ gets updated you can use serverless options like cloud run. It’s a simple process whenever BQ gets updated logs will be generated and we can create cloud run triggers based on the logs generated. The entire process of implementation is explained in this documentation please check.