Receiving SQS Messages Very Slowly-CodePudding

I am attempting parallel processing with Lambda. I'm invoking about 150 Lambda functions at nearly the same time. Each of these Lambda's are sending their status to the same standard SQS queue. Each is sending 10 status messages (i.e., 10, 20, 30,..., 100), so only about 1,500 messages total. I'm using the average of these statuses to display the overall status to the user.

I see evidence that all the Lambdas are finishing their job in about 10 seconds, but many of the messages are taking several minutes to arrive, so the user is still waiting even though the work is done.

The receiver of the messages is a PHP script, and here's the receiveMessage code:

while (!$done) {
    try {
        $result = $sqsClient->receiveMessage(array(
            'AttributeNames' => ['SentTimestamp'],
            'MaxNumberOfMessages' => 10,
            'WaitTimeSeconds' => 5,
            'MessageAttributeNames' => ['All'],
            'QueueUrl' => '<my queue>'
            ));

        foreach ($result->search('Messages[]') as $message) {
            ...

It's the same speed even if WaitTimeSeconds is 0 (short polling).

In the Console, Delivery Delay is set to 0.

What am I doing wrong? Is there a way to configure or code to get the messages almost as fast as they are sent? Thanks!

CodePudding user response：

SQS is a pull-based queue. You set WaitTimeSeconds to 0, that means the response from SQS will be returned without delay. But you are sending requests to SQS periodically, and if a response is empty AWS clients backs off, and waits longer each time.

If you need an event immediately, you should a push-based queue. On AWS it's SNS. You subscribe to an SNS topic and it sends a message as soon as it gets it.