I am using DynamoDB, and although there is a DAX cluster associated with this database, I am looking to do some scans just on the underlying database for now. (I am running this code locally with session-based AWS auth, and DynamoDB is available in the default VPC, but DAX appears to be in a private VPC, so I can't reach that for now. I will treat that as a separate problem).

I want to run a scan that gets all records. There are about 4k rows in the table, and I am only getting about 1.2K rows, which I put down to DynamoDB's design, which can only serve 1M per page. So I have created a loop, but it seems to be looping in the pagination, and going back to the start.

Here is some code:

interface CreditKey {
  emailAddress: string;
  shopperGroupId: number;
}

const options = {
  endpoint: 'http://dynamodb.eu-west-1.amazonaws.com',
};
const dynamoDbServiceClient = new DynamoDB(options);
const dynamoDbDocumentClient = new DynamoDB.DocumentClient(options);

let nextPage: DynamoDB.Key | undefined;
let rows: CreditKey[] = [];
let page: DocumentClient.ScanOutput;

console.log(`** Start`);
// Ooh, so DynamoDB won't serve more than 1MB per scan, so we if want
// the lot, we have to keep asking for it in pages!