How many simultaneous fetch calls can a lambda make-CodePudding

I've got a NodeJS14 Lambda on AWS whose purpose is to go out to various device endpoints and pull in data. It runs on a schedule.

The idea is I'd just loop through my needs and do an 'fetch' call for each.

Something like (psudocode):

 for (device of deviceList) {
     fetch(device.uri)
         .then(resp => saveToS3(resp) )
         .catch(err => notifyAdminOfError(err) ) 
 }

How many of these fetch calls can I run at a time? I expect my deviceList to get to around 10k long-term but will likely start around 1k.

If there is a limit (either in lambda, in node.js, or just a practical limit), I can break it up with a 'master' that hands out 100 devices (or whatever the limit) to a multiple secondary lambdas as needed... but I'd rather not complicate things like that needlessly.

CodePudding user response：

Lambdas are essentially single-core so having multiple threads may not provide any benefit (Reference: AWS Forum response)

There is also a function timeout limit of 15 minutes (Lambda limits)

The answer to your question depends on how long your fetch call takes. If it is a few milliseconds, then you can process 1k devices using a single Lambda under 15 minutes. If it is longer, you will need to process them in batches. You can use the same Lambda and call it with different batch inputs to process all devices.

CodePudding user response：

There is a concurrency Quota limit inside the lambda function, but it is for the number of calls you made to your Lambda function Invocation Scaling.

Burst concurrency quotas

3000 – US West (Oregon), US East (N. Virginia), Europe (Ireland)
1000 – Asia Pacific (Tokyo), Europe (Frankfurt), US East (Ohio)
500 – Other Regions

But if I’m right your question is related to the concurrent calls inside the lambda function.

In lambda there’s a way of making the async call where the Event queue handles all the incoming requests, that’s one of doing it.

But it comes with other setbacks as well like eventually consistent event queue Invocation async

Another solution that I can think of is by “decoupling” your reading device call with AWS SQS but it could be overkill for your use case.