Parallel lambdas run slowly-CodePudding

I'm trying to run a process launching 1000 lambdas in parallels.

I made a test with 10 lambdas that succeeded in ~4 seconds, however scale up to 1000 lambdas increase the running time to ~23 seconds.

const lambda = new AWS.Lambda();
await Promise.all(
    Array(1000)
      .map((_, i) => {
        if (i === 0) console.log("start lambda", i, new Date().getSeconds());
        if (i === 999) console.log("start lambda", i, new Date().getSeconds());
        return lambda
          .invoke({
            FunctionName: "myFunction",
            Payload: JSON.stringify(payload),
          })
          .promise()
          .then((x) => {
            if (i === 0) console.log("end lambda", i, new Date().getSeconds());
            if (i === 999) console.log("end lambda", i, new Date().getSeconds());
            return x;
          });
      })
);

output

start lambda 0 11
start lambda 999 11
end lambda 0 15
end lambda 999 39

I'm quite surprised about that observation because Lambda is a service designed to be highly scalable. I'd like to understand why is it so slower and what could I do to solve this problem.

CodePudding user response：

I think you are being limited by the SDK. Nothing to do with Lambda.

By default, the SDK allow 50 requests to run in parallel.

In your code, you have to change the maxSockets to something that works for you.

Something like this:

var AWS = require('aws-sdk');
var https = require('https');
var agent = new https.Agent({
   maxSockets: 1000
});

const lambda = new AWS.Lambda({
   apiVersion: '2012-08-10'
   httpOptions:{
      agent: agent
   }
});

But in my opinion, it is not a good way to test your lambda function execution. Using Lambda monitor, Lambda Insights and AWS X-Ray is a better way to look at it.

Alternatively, I will encourage you to deploy this solution to see if your lambda can be fined tuned: https://github.com/alexcasalboni/aws-lambda-power-tuning

It is not because you chose the 128MB memory that your lambda will be cheaper, you can be very surprised by the results with higher memory.