Background: We have an S3 bucket (Bucket A) publishing event notifications to an SQS queue on object creation.

For large SQS messages, AWS recommends using the SQS Extended Client Library (refer this article) which bootstraps a completely separate S3 bucket (Bucket B) and passes the reference to that object in Bucket B to the SQS queue.

My questions is: How is this different from simply passing a reference to the object in Bucket A to the SQS queue for all messages, regardless of size? I'm not convinced using a completely separate Library with a completely separate S3 bucket referenced is the best approach for this use case.

Thank you!

CodePudding user response：

First, I'd like to point out that you have S3 publishing events to an SQS queue. S3 is never going to use the extended client library to publish those events. S3 only publishes JSON metadata about the S3 object to the queue, so the messages themselves are always small (small enough to fit in SQS). In this scenario you have no use for the extended client library.

You would only need to consider using the extended client library if you want to post actual messages directly to SQS, but those messages can sometimes be too large to fit directly in the SQS queue. In that specific scenario, the extended client library would handle automatically uploading the message contents to S3 when publishing messages to the queue, and automatically reading the message contents from S3 when consuming messages from the queue.

That's all the extended library does, and if you are comfortable writing the S3 code yourself, there isn't anything else the extended library is going to do for you.

Also note that the extended library is only available for Java, so if you wanted to use a non-JVM programming language to interact with the queue you wouldn't be able to use it anyway.