Our Google Analytics data events are exported to BigQuery tables. I have reports that need to run when the events data arrives which are set up as AWS lambdas with python code (for various reasons and I can't immediately move these to be Google Cloud Functions etc).

Is it possible to have the creation of a table trigger a lambda? At present, I have a lambda periodically checking to see if the table has been created which seems suboptimal. Eventarc looks like it might possibly be the way to monitor for the creation event at the BigQuery end but it doesn't seem obvious how you'd interface with AWS.

Any genius ideas? I have dug repeatedly through StackOverflow, but can't see a match for this issue

CodePudding user response：

Eventarc isn't magic, it's only a wrapper of different things that you can do and customize (with a custom destination and not a Cloud Run).

Typically, Eventarc do:

• Create a Cloud Logging sink on a specific log filter (filter what you want to get custom events)
• Sink the filtered log entries in PubSub topic
• Create a PubSub push subscription that invoke Cloud Run HTTP endpoint.

You can create piece by piece all those steps. And in the latest one, invoke your AWS Lambda instead of Cloud Run.

But the difficulty is not here. The difficulty comes from the variety of table creation possibilities:

• By API call (table creation API)
• By Load Job (load a file into a table create it automatically but without invoking the table creation API)
• Directly in SQL with CREATE TABLE statement (but you can have also this statement in a script, you can have dynamic SQL,...)

And you might want to capture also the other creations (views, materialized views, procedure, functions,....)

At the end, your current method (invoque periodically the schema metadata info and get the recent addition in a dataset) could be the most "effortless" efficient!