How to set retry delay options for DynamoDB using Boto3 with Python?-CodePudding

I'm trying to avoid the ProvisionedThroughputExceededException by setting a custom exponential backoff using the "base" option like we can do in JavaScript according to this answer:

AWS.config.update({
  maxRetries: 15,
  retryDelayOptions: {base: 500}
});

As it's explained in this documentation, the "base" parameter defines the number used to increase the delay, so "base: 500" would increase the delay like this: 500, 1000, 1500, ...

I'm trying to make the same settings with Boto3 using Python 3.8, but the Boto3 Documentation seems to only allow setting the maximum retries attempts, not the delay options:

from botocore.client import Config

config = Config(
   retries = {
      'max_attempts': 10,
      'mode': 'standard'
   }
)

The "mode" option only gets three values: "legacy", "standard" and "adaptive". The doc also mentions a _retry.json file where these options are described and it looks like the "base" option is hardcoded in this file:

"dynamodb": {
      "__default__": {
        "max_attempts": 10,
        "delay": {
          "type": "exponential",
          "base": 0.05,
          "growth_factor": 2
        }

So, my question is: Is there any way to set the exponential backoff using Boto3 with Python for DynamoDB?

CodePudding user response：

This is hardcoded for Boto3 unfortunately & there is no way to modify the base retry delay using the Python SDK. Unless you write your own wrapper around the SDK calls, this isn't possible out of the box.

It may be worth to create an issue in the public repository for it to be picked up, or contribute directly.

CodePudding user response：

As mentioned by @Ermiya, I had to implement it myself. I didn't want to modify the boto3 default settings, I preferred to create a workaround inside the paginator, like this:

if page.get('RetryAttempts') and page.get('RetryAttempts') != 0:
        sleep(10 * page.get('RetryAttempts'))

So the code ended up like this:

from boto3.dynamodb.types import TypeDeserializer
td = TypeDeserializer()

def get_all_items(db_table):
    q_params = {
        'TableName': db_table.name,
    }
    total = 0
    paginator = db_client.get_paginator('scan')
    for page in paginator.paginate(**q_params):

        # Here's the workaround
        if page.get('RetryAttempts') and page.get('RetryAttempts') != 0:
            time.sleep(10 * page.get('RetryAttempts'))

        for db_item in page.get('Items'):
            db_item = {k: td.deserialize(v) for k, v in db_item.items()}
            yield db_item
        total  = page.get('Count')
        print(f"{total=}", end='\r')
    print(f"{total=}")

Of course, you can modify the values I used in time.sleep(10 ...) and tune it as you wish ;)