The use case here is I have to register students for a course . The input to this lambda will be student ID and Course ID. It involves following steps.
- call API 1 to get Student info
- call API 2 to get Course info
- Store Student details first in API 3 and get unique ID
- Save the ID and status as "Student_Registered" in DB (UI will keep checking status of registration process)
- Store StudentCourse details in API 3 with the unique ID generated in step 3 .
- Save the Registed object with status "Registration Completed" in DB
In this cases there might be possibility to have failure happens
- delay / exception from API 1
- delay / exception from API 2
- delay / exception from API 3 while storing Student
- exception from API 3 while storing Course (Note Student is registered and StudentCourse is failed )
- Save in DB may fail
My thoughts on this is to update DB after every step so that if At any point exception occurs I should be able to retry from the step it failed . What would be the efficient/best way to implement this. I read about step function would be the best way to handle this . Need suggestion from experienced people.Any examples / reference would also help.
CodePudding user response：
Whether you implement this as a step function or as a single Lambda function the issues still remain, Idempotency.
If you have a partial failure I would recommend signalling to the calling system that there has been a retry-able failure. This could be with a http status code or your own API contract.
The calling system should retry the request and the process should start again. The parts of the system would write or change data should be idempotent. If they receive a duplicate request they should not duplicate the action.
How does something know if a request is a duplicate or not? The calling system should send a idempotency key. A value that is same when retrying the same action, but otherwise unique. A UUID or a hash of the request are examples.
Idempotency isn't trivial to implement, it requires your system to hold state about the requests they've received.
In the AWS Lambda space the Lambda Powertools project has produced client libraries to help with Idempotency. You can find implementations in https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/idempotency/ and Java.
CodePudding user response：
One approach to be explored is solve your problem with SQS. You can attach lambda functions to it (triggers) and use the AWS Lambda Powertools to make you life even easier to process the messages.
If some error is raised in the process, the message will hit the lambda more than one (kind of retries) and if still not successful, your message will ends in a Dead-Letter queue (error handling). Using 2 or 3 queues you might get the job done.
You can even use FIFO queues with de-duplication.
Use the approach of "upsert" when saving the data, then the "retries" doesn't duplicate it.