What is the meaning of [request.ContinuationToken = response.NextContinuationToken] in the code belo-CodePudding

I am trying to get a list of objects inside given Amazon S3 client's bucket and prefix. I was suggested to use the following code, and it seems to be working well, but I don't understand why. I am really not familiar with AWS and S3 at all. Could someone make it clear for me to understand the meaning of ContinuationToken, NextContinuationToken and the meaning of this expression

request.ContinuationToken = response.NextContinuationToken;

in the code below please?

Thanks in advance!

    ListObjectsV2Request request = new ListObjectsV2Request
        {
            BucketName = bucketName,
            Prefix= prefixName
        };
    ListObjectsV2Response response; 
    do
        {
            response = await client.ListObjectsV2Async(request);

            // Process the response.
            foreach (S3Object entry in response.S3Objects)
            {
                Console.WriteLine("key = {0} size = {1}",
                    entry.Key, entry.Size);
            }
            Console.WriteLine("Next Continuation Token: {0}", response.NextContinuationToken);
            request.ContinuationToken = response.NextContinuationToken;
        } while (response.IsTruncated);

CodePudding user response：

A lot of AWS' apis are paged meaning they only return e.g. 1000 results and if you want more results you need a second api call where you tell them you want the next 1000 results.

The way to tell the API "I want the next 1000" results is by specifying a NextToken, a PageToken, a ContinuationToken, ... The name of the token differs between APIs but the concepts stays the same.

What the code does is that is mutates the request, sets the ContinuationToken to the token the API currently responded with. That way during the next loop iteration the API knows what 1000 next results to send you. And if there are even more results to fetch the API will respond with a different NextContinuationToken. Rinse and repeat until the API finally sets the IsTruncated flag to false meaning that there are no more results to fetch.

CodePudding user response：

Let's have a look at the docs. A quick search for "S3 NextContinuationToken" yields this page:

NextContinuationToken

NextContinuationToken is sent when isTruncated is true, which means there are more keys in the bucket that can be listed. The next list requests to Amazon S3 can be continued with this NextContinuationToken. NextContinuationToken is obfuscated and is not a real key.

So you've got a bucket with more keys that can fit in a single response. You need to make multiple requests to fetch all of the keys. Each response gives you a token indicating where that response finished, and where the next response should begin.

By passing NextContinuationToken to the next request, you're telling the AWS API that you'd like the next response to pick up where the last response left off.