Amazon DynamoDB taking long time to fetch record with high latency up to 5-6 second-CodePudding

pom.xml

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-bom</artifactId>
    <version>1.11.256</version>
    <type>pom</type>
    <scope>import</scope>
</dependency>
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-dynamodb</artifactId>
</dependency>

Client Code

public static AmazonDynamoDB getDynamoDBClient() {
        return AmazonDynamoDBClientBuilder.standard().withRegion("ap-southeast-1").build();
}

Now i am trying to execute a normal query having few records but it is taking long time to fetch the result.

For first time it is fetching the records in around 5-6 seconds on multiple requests in reduces by half. 2-3 seconds is still large time for fetching only few items.

Already checked the tuning of dynamo DB using different client configurations (connection timeout, request timeout, retry etc.) but not giving results as expected. Also checked with SDK version 2 URLConnectionHTTPClient config but same results came there too.

One possible cause can be the credentials fetch time for dynamo DB client but not having any credentials caching reference in java. Can any one suggest possible configuration to improve this latency.

CodePudding user response：

You are using a very old API and is not best practice anymore. To use best practice with Java, use the AWS SDK for Java v2.

The AWS SDK for Java 2.x is a major rewrite of the version 1.x code base. It’s built on top of Java 8 and adds several frequently requested features. These include support for non-blocking I/O and the ability to plug in a different HTTP implementation at run time.

The POM for v2 is:

 <dependency>
  <groupId>software.amazon.awssdk</groupId>
  <artifactId>bom</artifactId>
  <version>2.17.190</version>
  <type>pom</type>
  <scope>import</scope>
 </dependency>

To retrieve records using the AWS SDK for Java v2, you have three choices.

Use the DynamoDbClient.
Use the Enhanced Client (that maps Java objects to tables)
Use PartiQL (uses SQL like syntax)

I will show you all ways. All examples can be found in the Amazon Github repo.

Use the DynamoDbClient

Code is:

package com.example.dynamodb;

// snippet-start:[dynamodb.java2.get_item.import]
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
import software.amazon.awssdk.services.dynamodb.model.GetItemRequest;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
// snippet-end:[dynamodb.java2.get_item.import]


/**
 * Before running this Java V2 code example, set up your development environment, including your credentials.
 *
 * For more information, see the following documentation topic:
 *
 * https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
 *
 * To get an item from an Amazon DynamoDB table using the AWS SDK for Java V2, its better practice to use the
 * Enhanced Client, see the EnhancedGetItem example.
 */
public class GetItem {

    public static void main(String[] args) {

        final String usage = "\n"  
                "Usage:\n"  
                "    <tableName> <key> <keyVal>\n\n"  
                "Where:\n"  
                "    tableName - The Amazon DynamoDB table from which an item is retrieved (for example, Music3). \n"  
                "    key - The key used in the Amazon DynamoDB table (for example, Artist). \n"  
                "    keyval - The key value that represents the item to get (for example, Famous Band).\n" ;

        if (args.length != 3) {
            System.out.println(usage);
            System.exit(1);
        }

        String tableName = "Customer" ; //args[0];
        String key = "id" ; //args[1];
        String keyVal = "id101" ; //args[2];
        System.out.format("Retrieving item \"%s\" from \"%s\"\n", keyVal, tableName);

        ProfileCredentialsProvider credentialsProvider = ProfileCredentialsProvider.create();
        Region region = Region.US_EAST_1;
        DynamoDbClient ddb = DynamoDbClient.builder()
                .credentialsProvider(credentialsProvider)
                .region(region)
                .build();

        getDynamoDBItem(ddb, tableName, key, keyVal);
        ddb.close();
    }

    // snippet-start:[dynamodb.java2.get_item.main]
    public static void getDynamoDBItem(DynamoDbClient ddb,String tableName,String key,String keyVal ) {

        HashMap<String,AttributeValue> keyToGet = new HashMap<>();
        keyToGet.put(key, AttributeValue.builder()
                .s(keyVal).build());

        GetItemRequest request = GetItemRequest.builder()
                .key(keyToGet)
                .tableName(tableName)
                .build();

        try {
            Map<String,AttributeValue> returnedItem = ddb.getItem(request).item();

            if (returnedItem != null) {
                Set<String> keys = returnedItem.keySet();
                System.out.println("Amazon DynamoDB table attributes: \n");

                for (String key1 : keys) {
                    System.out.format("%s: %s\n", key1, returnedItem.get(key1).toString());
                }
            } else {
                System.out.format("No item found with the key %s!\n", key);
            }
        } catch (DynamoDbException e) {
            System.err.println(e.getMessage());
            System.exit(1);
        }
    }
    // snippet-end:[dynamodb.java2.get_item.main]
}

Enhanced Client

Code is:

package com.example.dynamodb;

// snippet-start:[dynamodb.java2.mapping.getitem.import]
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable;
import software.amazon.awssdk.enhanced.dynamodb.Key;
import software.amazon.awssdk.enhanced.dynamodb.TableSchema;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
// snippet-end:[dynamodb.java2.mapping.getitem.import]

/*
 * Before running this code example, create an Amazon DynamoDB table named Customer with these columns:
 *   - id - the id of the record that is the key
 *   - custName - the customer name
 *   - email - the email value
 *   - registrationDate - an instant value when the item was added to the table
 *
 * Also, ensure that you have set up your development environment, including your credentials.
 *
 * For information, see this documentation topic:
 *
 * https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
 */

public class EnhancedGetItem {

    public static void main(String[] args) {

        ProfileCredentialsProvider credentialsProvider = ProfileCredentialsProvider.create();
        Region region = Region.US_EAST_1;
        DynamoDbClient ddb = DynamoDbClient.builder()
                .credentialsProvider(credentialsProvider)
                .region(region)
                .build();

        DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder()
                .dynamoDbClient(ddb)
                .build();

        getItem(enhancedClient);
        ddb.close();
    }

    // snippet-start:[dynamodb.java2.mapping.getitem.main]
    public static void getItem(DynamoDbEnhancedClient enhancedClient) {

        try {
            DynamoDbTable<Customer> table = enhancedClient.table("Customer", TableSchema.fromBean(Customer.class));
            Key key = Key.builder()
                    .partitionValue("id120")
                    .build();

            // Get the item by using the key.
            Customer result = table.getItem(r->r.key(key));
            System.out.println("******* The description value is " result.getCustName());

        } catch (DynamoDbException e) {
            System.err.println(e.getMessage());
            System.exit(1);
        }
    }
    // snippet-end:[dynamodb.java2.mapping.getitem.main]
}

PartiQL

Code is

public static void getItem(DynamoDbClient ddb) {

        String sqlStatement = "SELECT * FROM MoviesPartiQ where year=? and title=?";
        List<AttributeValue> parameters = new ArrayList<>();
        AttributeValue att1 = AttributeValue.builder()
                .n("2012")
                .build();

        AttributeValue att2 = AttributeValue.builder()
                .s("The Perks of Being a Wallflower")
                .build();

        parameters.add(att1);
        parameters.add(att2);

        try {
            ExecuteStatementResponse response = executeStatementRequest(ddb, sqlStatement, parameters);
            System.out.println("ExecuteStatement successful: "  response.toString());

        } catch (DynamoDbException e) {
            System.err.println(e.getMessage());
            System.exit(1);
        }
    }

All of these ways are recommend using over AWS SDK for Java V1. If you are not familiar with the V2 API - including creds and setting up your dev environment, see:

Developer guide - AWS SDK for Java 2.x

CodePudding user response：

For first time it is fetching the records in around 5-6 seconds on multiple requests in reduces by half. 2-3 seconds is still large time for fetching only few items.

This is expected even with dynamo at production because first time you fetch the following sequence of operations happens:-

Look up for the partition ID your requested data belongs to from the route table.
Caches the Key to Partiton
Then refers to the partition for fetching the data.

As you see there is caching in step 2, so the next time requests go, the partition id is fetched from the cache thus the latency decreases. (will share the source once I have it)

Can any one suggest possible configuration to improve this latency.

Also, please don't use local dynamo for performance benchmarking, because it internally uses SQLite, i.e. SQL DB for storing. refer:- https://www.dbi-services.com/blog/aws-dynamodb-local/#:~:text=Yes, this NoSQL database is,Local engine, embedded in Java.