I created an on-demand DynamoDb table, and as I know that Dynamodb automatically scales the write/read capacity on on-demand mode.

But AWS Glue job gives error as "An error occurred while calling o201.pyWriteDynamicFrame. DynamoDB write exceeds max retry 10" because of the write capacity. How is this possible if the table is on on-demand mode? I didn't set any read/write capacity and and the table isn't even on the provisioned mode.

Dynamodb Table:

AWS Glue job output:

Dynamodb Tabble Throttled:

Thanks.

CodePudding user response：

Here is what you need to know about On-Demand mode tables

On Demand

If you recently switched an existing table to on-demand capacity mode for the first time, or if you created a new table with on-demand capacity mode enabled, the table has the following previous peak settings, even though the table has not served traffic previously using on-demand capacity mode:

Following are examples of possible scenarios.

• A provisioned table configured as 100 WCU and 100 RCU. When this table is switched to on-demand for the first time, DynamoDB will ensure it is scaled out to instantly sustain at least 4,000 write units/sec and 12,000 read units/sec.

• A provisioned table configured as 8,000 WCU and 24,000 RCU. When this table is switched to on-demand, it will continue to be able to sustain at least 8,000 write units/sec and 24,000 read units/sec at any time.

• A provisioned table configured with 8,000 WCU and 24,000 RCU, that consumed 6,000 write units/sec and 18,000 read units/sec for a sustained period. When this table is switched to on-demand, it will continue to be able to sustain at least 8,000 write units/sec and 24,000 read units/sec. The previous traffic may further allow the table to sustain much higher levels of traffic without throttling.

• A table previously provisioned with 10,000 WCU and 10,000 RCU, but currently provisioned with 10 RCU and 10 WCU. When this table is switched to on-demand, it will be able to sustain at least 10,000 write units/sec and 10,000 read units/sec.

Important

If you need more than double your previous peak on table, DynamoDB automatically allocates more capacity as your traffic volume increases to help ensure that your workload does not experience throttling. However, throttling can occur if you exceed double your previous peak within 30 minutes. For example, if your application’s traffic pattern varies between 25,000 and 50,000 strongly consistent reads per second where 50,000 reads per second is the previously reached traffic peak, DynamoDB recommends spacing your traffic growth over at least 30 minutes before driving more than 100,000 reads per second.

Above information is directly from AWS Docs src

Glue Workers

Now, when you begin to write using AWS Glue, you will very quickly exceed the 4000 WCU limit, which means you have exceeded the rule which is double your previous peak (4000) within 30 minutes.... So what now??

Pre-warming your table

DynamoDB provides you capacity in the form of partitions, where each partition is capable of providing you 1000 WCU and 3000 RCU. DynamoDB only ever scales partitions out, never merging in.

For that reason, we can "pre-warm" our DynamoDB tables by creating them in Provisioned-mode and allocating our peak WCU. For eg. let's imagine we expect Glue to consume 40,000 WCU, then we will be sure our table can handle that following these steps:

• Create table in provisioned mode
  • No Autoscaling
  • 40,000 WCU
  • 40,000 RCU
• When table is marked as Active (1-2 mins)
  • Switch capacity mode to On-Demand

Now, you have a new DynamoDB table in On-Demand which is capable of providing 40,000 WCU out of the gates, not the 4,000 WCU provided by default. This will eliminate throttling from Glue.

CodePudding user response：

DynamoDB sets read/write capacity for its on-demand tables in order to balance performance and cost. The read/write capacity units determine the rate at which DynamoDB can read and write data to the table, with a larger number of units allowing for a higher rate of read/write operations. By setting these values, users can control the performance of their DynamoDB table and ensure that it meets the demands of their application. Additionally, setting the capacity units helps DynamoDB automatically manage the distribution of data and traffic, ensuring low latency and high reliability.