Limit 1000 Records from CosmosDB Query-CodePudding

I need to limit the results of a query to CosmosDB to 1000 records, I am trying to set the feed iterator to exit once the result list hits 1000 but right now it is stopping after 100 records. Here is my current code:

        public async Task<IEnumerable<AS2InboundLogRecord>> RetrieveInboundLogs(string partitionKeyName, DateTime startDate, DateTime endDate)
        {
            var inboundLogs = new List<AS2InboundLogRecord>();
            string continuationToken = null;
            int itemLimit = 1000;

            QueryRequestOptions requestOptions = new QueryRequestOptions()
            {
                MaxItemCount = 100
            };


            using (FeedIterator<AS2InboundLogRecord> setIterator = dbContainer.GetItemLinqQueryable<AS2InboundLogRecord>(true, continuationToken, requestOptions)
                    .Where(x => (x.PartitionKey == partitionKeyName || partitionKeyName == null) &&
                      (x.MessageDate >= startDate) &&
                      (x.MessageDate <= endDate))
                    .ToFeedIterator())
            {
                while (setIterator.HasMoreResults)
                {
                    FeedResponse<AS2InboundLogRecord> response = await setIterator.ReadNextAsync();
                    inboundLogs.AddRange(response);

                    if (response.Count >= itemLimit) { break; }
                }

                Console.WriteLine(inboundLogs.Count());
                return inboundLogs.OrderByDescending(x => x.MessageDate);
            };
        }

Any input would be appricated, thanks

CodePudding user response：

I think you have to correct two things:

First: you set limitCount to 100 sound like it just fetch only 100 records , if this limit fetch count set it to 1000 otherwise goto Second phrase.

Second: your if condition may not work because you compare response.Count while inboundLogs.Count should be compare.

Correction:

public async Task<IEnumerable<AS2InboundLogRecord>> RetrieveInboundLogs(string partitionKeyName, DateTime startDate, DateTime endDate)
{
    var inboundLogs = new List<AS2InboundLogRecord>();
    string continuationToken = null;
    int itemLimit = 1000;

    QueryRequestOptions requestOptions = new QueryRequestOptions()
    {
        MaxItemCount = 10000
    };


    using (FeedIterator<AS2InboundLogRecord> setIterator = dbContainer.GetItemLinqQueryable<AS2InboundLogRecord>(true, continuationToken, requestOptions)
            .Where(x => (x.PartitionKey == partitionKeyName || partitionKeyName == null) &&
                (x.MessageDate >= startDate) &&
                (x.MessageDate <= endDate))
            .ToFeedIterator())
    {
        while (setIterator.HasMoreResults)
        {
            FeedResponse<AS2InboundLogRecord> response = await setIterator.ReadNextAsync();
            inboundLogs.AddRange(response);

            if (inboundLogs.Count >= itemLimit) { break; }
        }

        Console.WriteLine(inboundLogs.Count());
        return inboundLogs.OrderByDescending(x => x.MessageDate);
    };
}

As cosmosdb doesn't support Skip and Take you can also use offset and limit described here or you can also use ContinuationToken of response to make sure there is still item, something like:

do
{ 
    response = await setIterator.ReadNextAsync();
    //List<dynamic> documents = response.Resource.ToList();
    continuationToken = response.ContinuationToken;
    inboundLogs.AddRange(response);
} while (continuationToken != null && inboundLogs.Count <= itemLimit);