I have multiple Redshift stored procedures (~15), some are dependent on the previous run stored procedures while some can run asynchronously.

I need to orchestrate this with proper failure handling in case any successor stored procedure fails then I can run that particular one.

I tried orchestrating this using AWS Eventbridge but in that, I found many limitations. Like triggering any specific stored procedure. Using Eventbridge rules 5 targets to run both combination sync and async

Is there any way to run my stored procedures in AWS Glue using custom canvas, to construct orchestration. Putting one stored procedue in one block

How to make redshift connection in the flow diagram so that my stored procedure will be executed in my redshift cluster?

CodePudding user response：

I don't know about Glue (I doubt it) but this is a great case for Step Functions. Lots of blogs about how to set up a serverless data orchestration process using Step Functions with proper error / exception handling. A place to start - https://aws.amazon.com/blogs/big-data/orchestrate-multiple-etl-jobs-using-aws-step-functions-and-aws-lambda/