How to handle "The given header was not found" when paging records in c# API GET request?-CodePudding

I'm requesting data from an API that requires paging records based on a custom header called "cursor". Only 100 records may be retrieved per call and as such I've created a while loop to execute. The loop functions... until it doesn't. Once all records are paged, the headers get dropped and my program errors out with "The given header was not found." No inserts into my database occur because the requests are streamed until they are all paged.

How can I handle this so the program completes successfully?

The call being made is:

var products = await ProcessProducts(userAndPasswordToken, BaseUrl);

The task being called:

    private static async Task<List<Products>> ProcessProducts(string userAndPasswordToken, string BaseUrl)
    {
        //Construct urls
        string RequestPath = string.Format("food/products");
        string FullUrl = string.Format("{0}{1}", BaseUrl, RequestPath);
        string CursorPath = string.Format("");

        //Use GetAsync instead of GetStreamAsync unless it's mandatory for your codebase.
        var response = await client.GetAsync(FullUrl);

        //Extract string from the response right away
        var content = await response.Content.ReadAsStringAsync();

        //Pass it in instead of Steam.
        var Products = Newtonsoft.Json.JsonConvert.DeserializeObject<List<Products>>(content);

        //Place the header key mapped with the cursor value.
        IEnumerable<string> values = response.Headers.GetValues("cursor");

        // string cursor = null;
        string cursor = "4146";
        // cursor = values?.FirstOrDefault();
        do
        {

            CursorPath = $"?cursor={cursor}";
            if (cursor == null) CursorPath = string.Format("");
            FullUrl = string.Format("{0}{1}{2}", BaseUrl, RequestPath, CursorPath);
            response = await client.GetAsync(FullUrl);
            content = await response.Content.ReadAsStringAsync();
            var nextProducts = Newtonsoft.Json.JsonConvert.DeserializeObject<List<Products>>(content);

            // add Products in the next page.
            Products.AddRange(nextProducts);

            values = response.Headers.GetValues("cursor");
            cursor = values?.FirstOrDefault();

            System.Threading.Thread.Sleep(1000);
            Console.WriteLine(FullUrl);
            Console.WriteLine(response);

        } while (cursor != null);

        return Products;
    }

The insert happens here with a SQL stored procedure:

            using (SqlConnection conn = new SqlConnection(lConnString))
            {
                conn.Open();
                using (TransactionScope ts = new TransactionScope())
                {

                    foreach (var repo in products)
                    {
                        SqlCommand cmdIns = new SqlCommand("usp_insert_Products", conn);
                        cmdIns.CommandType = CommandType.StoredProcedure;
                        cmdIns.Parameters.AddWithValue("@ProductId", repo.ProductId.ToString() ?? (object)DBNull.Value);
                        cmdIns.Parameters.AddWithValue("@ProductDetailId", repo.ProductDetailId.ToString() ?? (object)DBNull.Value);
                        cmdIns.ExecuteNonQuery();
                    }

                    ts.Complete();
                }
                conn.Close();

The sample code I have been provided has a completely different method, which I am on the cusp of moving to if I can't get my code to function properly:

Sample code

// Paging is handled by making a request and then making
// follow up requests as long as a "cursor" is returned.
string cursor = null;
do
{
    var query = HttpUtility.ParseQueryString("");
    query["locationId"] = "1";
    query["businessDate"] = "2019-04-30";
    if (cursor != null) query["cursor"] = cursor;

    var fullUrl = $"{url}/{endpoint}?{query}";
    _testOutputHelper.WriteLine(fullUrl);
    var json = client.DownloadString(fullUrl);
    results.AddRange(
        JsonConvert.DeserializeObject<List<Check>>(json));

    cursor = client.ResponseHeaders["cursor"];
} while (cursor != null);

}

CodePudding user response：

You're using response.Headers.GetValues("cursor") which is documented to throw an InvalidOperationException if the header can't be found.

You should use TryGetValues instead:

if (response.Headers.TryGetValues("cursor", out IEnumerable<string> values))
{
    // header exists
}
else
{
    // header doesn't exist
}