I created a standard queue with the following specs:
- delay seconds: 0
- max receive count (redrive policy): 10
- visibility timeout: 300
I have a lambda consumer that receives the messages. From the documentation, it says that you need to explicitly delete the message in the consumer once it is processed (at least to guarantee that it is removed). However, it seems that the messages get deleted on their own regardless of them being deleted in the consumer or if the consumer returns an unsuccessful status code. The only way I've seen a message return to the queue is when I raise some sort of runtime error in the consumer.
Question: What is a proper error message to return to the consumer so that it knows to not delete the message?
CodePudding user response：
The Lambda service communicates with the queue, applying success and failure conditions to your function's return value. In a nutshell, the Lambda service interprets:
- Null or empty responses from your Lambda as 100% batch success
- Thrown errors or malformed responses as 100% batch failure
- Valid batchItemFailures in the response as partial failures
You can optionally manually delete messages from the queue with a DeleteMessage SDK call, although there is no obligation to do so. Manual deletion might make sense if your service is not idempotent and you really need to avoid double-processing.
CodePudding user response：
Quoting after AWS SQS visibility timeout documentation:
When a consumer receives and processes a message from a queue, the message remains in the queue. Amazon SQS doesn't automatically delete the message. Because Amazon SQS is a distributed system, there's no guarantee that the consumer actually receives the message (for example, due to a connectivity issue, or due to an issue in the consumer application). Thus, the consumer must delete the message from the queue after receiving and processing it.
Immediately after a message is received, it remains in the queue. To prevent other consumers from processing the message again, Amazon SQS sets a visibility timeout, a period of time during which Amazon SQS prevents other consumers from receiving and processing the message.
On your case, as the visibility is set to 300 seconds, the messages stay In flight mode, until the visibility period expires. On runtime errors, the message returns from inflight mode to available. This is the same as Negative acknowledgement behavior on Queue Services like RabbitMQ.
If the message isn't deleted, it will return back after the visibility period expires or an error occurs as the lambda hasn't fully processed the message.