I have project with the following workflow:

1. Pull files from server and upload to S3
2. When files hit S3, a message is sent to a topic using SNS
3. The lambda function subscribed to said topic will then process files by doing calculations

So far, I have not experienced any issues, but I wonder if this is a use case for SQS?

How many messages can my Lambda function handle all at once? If I am pulling hundreds of files, is SQS a necessity?

CodePudding user response：

By default, parallel invocation limit is set for 1000.

You can change that limitation, but i never hit that number so far.

As soon as a lambda is done with consuming current request, it will be used for another, so if you upload 1000 files, you probably will only need about 100 lambdas, unless you need Minutes for one lambda to run.

The AWS handles the queued triggers, so even if you upload 100.000 files, they will be consumed asap, depending on diverse criteria.

You can test it with creating many little files and upload them all at once :)

For higher speed, upload them to different bucket, and simply move from bucket to bucket (speed is higher this way)

gl !