How can I implement paging logic in C#?-CodePudding

I am requesting data from an API with built in paging. I am successfully able to request data (100 rows at a time) and insert it into my database. In cases where more than 100 rows are available, I am not getting the full dataset.

Basically paging is handled by making a request, and then making follow up requests as long as a "cursor" is returned.

I call the task with:

static async Task Main(string[] args){...

var employees = await ProcessEmployees(userAndPasswordToken, BaseUrl);

My task looks like this:

private static async Task<List<Employees>> ProcessEmployees(
    string userAndPasswordToken, string BaseUrl)
{
    //Construct urls
    string RequestPath = string.Format("/employees");
    string FullUrl = string.Format("{0}{1}", BaseUrl, RequestPath);

    var streamTask = client.GetStreamAsync(FullUrl);
    var employees = await JsonSerializer.DeserializeAsync<List<Employees>>(
        await streamTask);

    return employees;
}

How can I check the header for the cursor and then insert more than once if there are over 100 rows returned?

Sample code provided for paging by vendor:

// Paging is handled by making a request and then making
// follow up requests as long as a "cursor" is returned.
string cursor = null;
do
{
    var query = HttpUtility.ParseQueryString("");
    query["locationId"] = "1";
    query["businessDate"] = "2019-04-30";
    if (cursor != null) query["cursor"] = cursor;

    var fullUrl = $"{url}/{endpoint}?{query}";
    _testOutputHelper.WriteLine(fullUrl);
    var json = client.DownloadString(fullUrl);
    results.AddRange(
        JsonConvert.DeserializeObject<List<Check>>(json));

    cursor = client.ResponseHeaders["cursor"];
} while (cursor != null);

}

CodePudding user response：

Your question : How can I check the header for the cursor and then insert more than once if there are over 100 rows returned?

Code answer :

private static async Task<List<Employees>> ProcessEmployees(string 
userAndPasswordToken, string BaseUrl)
{
    //Construct urls
    string RequestPath = string.Format("/employees");
    string FullUrl = string.Format("{0}{1}", BaseUrl, RequestPath);
    
    // use GetAsync instead of GetStreamAsync unless it's mandatory for your codebase.
    var response = await client.GetAsync(FullUrl);

    // you can extract string from the response right away
    var content = await response.Content.ReadAsStringAsync();

    // and pass it in instead of Steam.
    var employees = await JsonSerializer.DeserializeAsync<List<Employees>>(content);

    // place the header key mapped with the cursor value.
    IEnumerable<string> values = response.Headers.GetValues("{your-cursor-key}");

    // check if `values` above has a value. If it returns more than one value,
    // you need to figure out how to get the right value.
    var cursor = values?.FirstOrDefault();
    while(cursor != null) 
    {
        // You haven't provided any information about what the cursor really is.
        // I just assumed that it's just a simple query string with `cursor` key name.
        RequestPath  = $"?cursor={cursor}";
        FullUrl = string.Format("{0}{1}", BaseUrl, RequestPath);
        response = await client.GetAsync(FullUrl);
        content = await response.Content.ReadAsStringAsync();
        var nextEmployees = await JsonSerializer.DeserializeAsync<List<Employees>>(content);

        // add employees in the next page.
        employees.AddRange(nextEmployees);
        
        values = response.Headers.GetValues("{your-cursor-key}");
        cursor = values?.FirstOrDefault();
    }

    return employees;
}

I just can't really give you an exact answer because there're so many missing details in your question. But it looks like you wanted to get some actual code snippets. The code snippet above cannot work and you need to modify it.

It entirely depends on how the server sends you a response with the Cursor.

Be aware that...

If there's always Cursor value in the response header, eg) "0" or "N/A", the cursor won't be null at all and your client program will keep running the while statement forever.
If you should use Cursor in request header when calling /employees, you should build HttpRequestMessage and use SendAsync.

Here's HttpRequestMessage and SendAsync example.

var requestMessage = new HttpRequestMessage
{
    Method = HttpMethod.Get,
    RequestUri = new Uri(FullUrl)
};
requestMessage.Headers.Add("{acceptable-cursor-header-key}", cursor);
response = client.SendAsync(requestMessage);

CodePudding user response：

With the help of @hina10531 I was able to rewrite the task with success. The issue I am now running into is too many requests (429) as my tier of access to this API only allows 60 calls per minute - another question, another post.

Answer below:

        private static async Task<List<Employees>> ProcessEmployees(string userAndPasswordToken, string BaseUrl)
    {
        //Construct urls
        string RequestPath = string.Format("general/employees");
        string FullUrl = string.Format("{0}{1}", BaseUrl, RequestPath);
        string CursorPath = string.Format("");
        
        // use GetAsync instead of GetStreamAsync unless it's mandatory for your codebase.
        var response = await client.GetAsync(FullUrl);

        // you can extract string from the response right away
        var content = await response.Content.ReadAsStringAsync();
        
        // and pass it in instead of Steam.
        //var employees = JsonSerializer.DeserializeAsync<List<Employees>>(content);
        var employees = Newtonsoft.Json.JsonConvert.DeserializeObject<List<Employees>>(content);
        
        // place the header key mapped with the cursor value.
        IEnumerable<string> values = response.Headers.GetValues("cursor");

        // check if `values` above has a value. If it returns more than one value,
        // you need to figure out how to get the right value.
       
        string cursor = null;
        cursor = values?.FirstOrDefault();
        do
        {
            // You haven't provided any information about what the cursor really is.
            // I just assumed that it's just a simple query string with `cursor` key name.
            CursorPath = $"?cursor={cursor}";
            if (cursor == null) CursorPath = string.Format("");
            FullUrl = string.Format("{0}{1}{2}", BaseUrl, RequestPath, CursorPath);
            response = await client.GetAsync(FullUrl);
            Console.WriteLine(response);
            content = await response.Content.ReadAsStringAsync();
            var nextEmployees = Newtonsoft.Json.JsonConvert.DeserializeObject<List<Employees>>(content);

            // add employees in the next page.
            employees.AddRange(nextEmployees);
            
            values = response.Headers.GetValues("cursor");
            cursor = values?.FirstOrDefault();

        } while(cursor != null);

        return employees;
    }