I've been trying to solve an issue I've encountered when downloading a batch of images in my app.

If I use HttpClient.GetStreamAsync(url) to download the batch, then it seems like some of the requests will timeout, and eventually error.

However if I use HttpClient.GetAsync(url), then the entire batch will download without any issue.

My suspicion is that it has something to do with ports not being freed up when invoking .GetStreamAsync(url), however I could be talking nonsense.

Below is a code snippet that demonstrates the issue.

async Task Main()
{   
    HttpClient httpclient = new HttpClient();
    var imageUrl = "https://tenlives.com.au/wp-content/uploads/2020/09/Found-Kitten-0-8-Weeks-Busy-scaled.jpg";
    var downloadTasks = Enumerable.Range(0, 15)
                                .Select(async u =>
                                {
                                    try
                                    {
                                        //Option 1) - this will fail
                                        var stream = await httpclient.GetStreamAsync(imageUrl);
                                        //End Option 1)

                                        //Option 2) - this will succeed
                                        //var response = await httpclient.GetAsync(imageUrl);
                                        //response.EnsureSuccessStatusCode();
                                        //var stream = await response.Content.ReadAsStreamAsync();
                                        //End Option 2)

                                        return stream;
                                    }
                                    catch (Exception e)
                                    {
                                        Console.WriteLine($"Error downloading image");
                                        throw;
                                    }
                                }).ToList();
    

    try
    {
        await Task.WhenAll(downloadTasks);
    }
    catch (Exception e)
    {       
        Console.WriteLine("================ Failed to download one or more image "   e.Message);
    }
    Console.WriteLine($"Successful downloads: {downloadTasks.Where(t => t.Status == TaskStatus.RanToCompletion).Count()}");
}

In the code block of the linq Select statement, Option 1) will fail as described above. If you comment out 1), and uncomment Option 2), then everything will succeed.

Can anybody explain what might be happening here?

EDIT: This seems to work ok with .net core. I can reproduce this issue using .net framework 4.7.2 and below

EDIT2: I've also observed that if I increase the default connection limit by adding ServicePointManager.DefaultConnectionLimit = 30; then the error no longer happens, however this does not explain why it fails for Option 1) but succeeds for Option 2)

CodePudding user response：

As explained by @RichardDeeming, HttpClient.GetStreamAsync call HttpClient.GetAsync with HttpCompletionOption.ResponseHeadersRead.

The code can be rewritten as follows:

async Task Main()
{   
    HttpClient httpclient = new HttpClient();
    var imageUrl = "https://tenlives.com.au/wp-content/uploads/2020/09/Found-Kitten-0-8-Weeks-Busy-scaled.jpg";
    var downloadTasks = Enumerable.Range(0, 15)
        .Select(async u =>
        {
            try
            {
                //Option 1) - this will fail
                var response = await httpclient.GetAsync(imageUrl, HttpCompletionOption.ResponseHeadersRead);
                //End Option 1)

                //Option 2) - this will succeed
                //var response = await httpclient.GetAsync(imageUrl, HttpCompletionOption.ResponseContentRead);
                //End Option 2)

                response.EnsureSuccessStatusCode();
                var stream = await response.Content.ReadAsStreamAsync();
                return stream;
            }
            catch (Exception e)
            {
                Console.WriteLine($"Error downloading image");
                throw;
            }
        }).ToList();

    try
    {
        await Task.WhenAll(downloadTasks);
    }
    catch (Exception e)
    {       
        Console.WriteLine("================ Failed to download one or more image "   e.Message);
    }
    Console.WriteLine($"Successful downloads: {downloadTasks.Where(t => t.Status == TaskStatus.RanToCompletion).Count()}");
}

Next HttpClient.GetAsync call HttpClient.SendAsync. We can see the method's code on GitHub :

//I removed the uninterested code for the question
public Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
{
    TaskCompletionSource<HttpResponseMessage> tcs = new TaskCompletionSource<HttpResponseMessage>();
    client.SendAsync(request, cancellationToken).ContinueWith(task =>
    {
        HttpResponseMessage response = task.Result;
        if(completionOption == HttpCompletionOption.ResponseHeadersRead)
        {
            tcs.TrySetResult(response);
        }
        else
        {
            response.Content.LoadIntoBufferAsync(int.MaxValue).ContinueWith(contentTask =>
            {
                tcs.TrySetResult(response);
            });
        }
    });
    return tcs.Task;
}

With HttpClient.GetAsync (or SendAsync(HttpCompletionOption.ResponseContentRead)), the received content in a network buffer is read and integrated in a local buffer.

I'm not sure about the network buffer, but I think a buffer somewhere (network card, OS, HttpClient, ???) is full and blocks new responses.

You can correct the code by correctly managing this buffer, for example by disposing the associated streams :

var downloadTasks = Enumerable.Range(0, 15)
.Select(async u =>
{
    try
    {
        var stream = await httpclient.GetStreamAsync(imageUrl);
        stream.Dispose(); //Free buffer
        return stream;
    }
    catch (Exception e)
    {
        Console.WriteLine($"Error downloading image");
        throw;
    }
}).ToList();

In .Net Core, the original code work without correction. The HttpClient class has been rewrited and certainly improved.