I am working on a backfill issue where I need to fetch all the unique values for an attribute in a dynamo db table and call a service to add these to the storage of that service. I am thinking of creating a temporary dynamo db table. I can read the original table in a lambda function and write only the unique values in the temp table. Is there any other approach possible?

The dynamo DB table has approximately 1,400,000 rows.

CodePudding user response：

1,400,000 records is not that many. Probably you can just read the table.

You can improve the read by making your attribute a global secondary key. It need not be unique. Then you can read only the attribute in question or check uniqueness.

If the records in your table are constantly updated, you can listen to the DynamoDB update stream and just update your temporary table with the new values.

Using the single table pattern https://www.youtube.com/watch?v=EOQqi6Yun7g - your "temporary" table can be just a different primary key prefix.

If you have to scan the table and the process is too long, you can split it to multiple lambda calls by passing around the LastEvaluatedKey value (e.g. with a step machine).

CodePudding user response：

1. You can scan the whole table, using projection expression fetch only the relevant columns and extract unique values.

2. One more approach can be, you can take a backup of DynamoDB table to S3 and then process the S3 file to extract unique column values.