Cloudformation failure due to kinesis rate limit exceeded-CodePudding

We have an AWS Lambda SAM template occasionally failing during Cloudformation stack creation/update due to:

Invalid request provided: Received Exception while reading from provided stream. Rate exceeded for shard shardId-000000000000

In the SAM template, 10 Lambda functions are attached (Kinesis events source for Lambda) to the same Kinesis Stream with 10 shards.

Does anybody what may be causing this issue? I don't see any limitation on the number of Lambda attached to a single Kinesis stream

CodePudding user response：

If this is happening when you update a stack, it's likely that you're running into one of the following quotas:

DescribeStream is limited to 10 transactions per second per account.
GetShardIterator is limited to 5 transactions per second per shard.

Both of these API calls need to happen for a Lambda to read from a stream, and almost certainly happen when the event trigger is created.

I'm not sure that there's a guaranteed solution with your current architecture, but one possibility is to use DependsOn to chain the creation of Lambdas and triggers. The time taken to create these resources should keep the API call rate below the limits.

If that doesn't work, you're going to have to use enhanced fan-out (which you might need anyway, depending on how much data is pushed over the stream). However, that won't solve the problem of DescribeStream quota.