WebClient download images in sequence causes issue with ImageList-CodePudding

I have spent a far bit of days getting into VB.NET and getting a program surprisingly fairly connected via MySQL & verifies bcrypt hashes via login dialog. All works so wonderfully.

When a user proceeds to Form2, we have this code to work the list from MySQL results:

Dim transferstable As New DataTable

sql = "SQL-SELECT-QUERY"

With cmd
    .Connection = con
    .CommandText = sql
End With
For Each row As DataRow In transferstable.Rows
    Using client As New WebClient()
        Dim Url = "domain.org/images/covers/" & row.Item("cover")
        client.DownloadFileAsync(New Uri(Url), "C:\\App_Uploads\" & row.Item("cover"))
    End Using
Next
For Each row As DataRow In transferstable.Rows
    ' myImage barely loading image for each row, usually first 2 rows out of ~160 rows
    Dim myImage As System.Drawing.Image = Image.FromFile("C:\\App_Uploads\" & row.Item("cover"))
    ListControl1.Add("somename", "item title", "description", "sideinfo", myImage, 1)
Next

So the first block works fine, it downloads ALL of the images to C:\\App_Uploads

However we can not properly pass these images to ListControl1.Add() without it giving up / running out of memory.

Using a fixed local image like C:\\test.png works fine and assigns to every single row in the list (all 160 rows) found from database, but how can we assign our offline (downloaded) images to the results? It is just wracking our brain on many hours now.

We got this far! thanks!

Result with images iterated locally per row:

Result with local image iterated per row:

Dim myImage As System.Drawing.Image = Image.FromFile("C:\\test.png")

UPDATE ---

We removed For Each from wrapping WebClient() and seem to be in the right direction, however only 1-2 images load into view.

' Download image art locally            
Using client As New WebClient()
    Dim Url = "domain.com/images/covers/" & transferstable.Rows(0).Item(14)
    Await client.DownloadFileTaskAsync(New Uri(Url), "C:\\App_Uploads\" & transferstable.Rows(0).Item(14))
End Using

For Each row As DataRow In transferstable.Rows
    Dim myImage As System.Drawing.Image = Image.FromFile("C:\\App_Uploads\" & row.Item("cover"))
    ListControl1.Add("somename", "item title", "description", "sideinfo", myImage, 1)
Next

CodePudding user response：

You're downloading Images in a loop using the event-driven (non-awaitable) version of WebClient's DownloadFile(), DownloadFileAsync(). The WebClient object is declared in with a Using statement.
Assuming that the WebClient instance can terminate the download before it's disposed, the DownloadFileAsync() method returns immediately: you're supposed to subscribe to the DownloadFileCompleted to receive a notification when the file is ready.

After this first loop completes, another loop is started, to retrieve the images from the disk.
Again assuming that all files are actually downloaded (I haven't tested it), you have a race condition, since you're trying to immediately use files that may not be completed or that are not actually there (and possibly never will be).

It's probably preferable, in this context, to download all the images in parallel, without storing them on disk, unless strictly required; only after the download is complete for sure, present the images in the UI.

An example of an alternative method that makes use of a static (Shared) HttpClient object declared in a helper class that exposes a public method, DownloadImages(), which is responsible for returning an ordered List(Of Image).
The HttpClient object is declared as Lazy(Of HttpClient) here.
See the documentation about the Lazy<T> class for more information.
(if you don't like it, you can probably change/remove the Lazy<T> instantiation here).

The order of the Images in the List depends on the order in which the download URLs are passed to the method: the images are returned in the same order.
The DownloadImages() method generates a numeric sequence, which is passed, along with the URL, to the private GetImage() method:

Dim tasks = urlList.Select(Function(url, idx) GetImage(idx, New Uri(url)))

The sequence is added because awating Task.WhenAll() doesn't guarantee that the Tasks are returned in a specific order.
The List(Of Task) is then reordered using the original sequence, in case the order is a matter of importance in your use case.

Note: the Images are downloaded in parallel. If you download a lot of images from the same address or quite frequently, your IP Address may end up in a black list.

To use the class, create a new DownloadImagesHelper object and a collection of Urls (as strings).
Then await the call to the DownloadImages() method.
When the method returns, loop the list of Images you get back and add new Items to your Control.
For example:

Dim urls As New List(Of String) 
For Each row As DataRow In transferstable.Rows
    urls.Add($"domain.org/images/covers/{row.Item("cover")}") 
Next

Dim downloadHelper = New DownloadImagesHelper()
Dim images = Await downloadHelper.DownloadImages(urls)

For Each img As Image In images
   ListControl1.Add("somename", "item title", "description", "sideinfo", img, 1)
Next

The class implements IDisposable. Call its Dispose() method when you're done with the DownloadImagesHelper object. The Dispose method tries to (it's not immediate) close existing connections and dispose of the HttpClient.

Helper class:

Note: the DownloadImages() doesn't check the status of the returned Tasks:

Return tasks.OrderBy(Function(t) t.Result.Key).Select(Function(t) t.Result.Value)

You may verify in different conditions whether it's preferable to add this check, as in:

Return tasks.Where(Function(t) t.Status = TaskStatus.RanToCompletion).
             OrderBy(Function(t) t.Result.Key).
             Select(Function(t) t.Result.Value)

In case you don't want to return a result when an Image could not be downloaded because of some HTTP exception, set the Image value to Nothing in GetImage() (described in code) and filter the results in DownloadImages():

Return tasks.OrderBy(Function(t) t.Result.Key).
             Where(Function(t) t.Result.Value IsNot Nothing).
             Select(Function(t) t.Result.Value)

Or just return the null results and filter later, in the code that called these methods.

Note: The HttpClientHandler is initialized setting its SslProtocols property.
This requires at least .Net Framework 4.8, otherwise it doesn't do anything (the Docs say .Net Framework 4.7.2 , don't believe it :)
Also, the property is hard-coded to SslProtocols.Tls12 (it's there to present the matter). You could remove it or set it to SslProtocols.Default, or add, e.g., SslProtocols.Tls11 in case the default System setting (usually TLS12) is not supported by one or more servers you're downloading from.
SslProtocols.Tls13, even though it's included, currently cannot be used, so don't add it.
If you're using an older .Net version, you need to set the TLS version using the ServicePointManager directly, before you create any connection. For example:

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12

EDIT:
Changed the return Type of DownloadImages() to Task(Of List(Of Image)) instead of Task(Of IEnumerable(Of Image)), since it may be simpler to debug and handle locally. Change it back if you need a deferred execution.

Added a DummyImage Property that can be set to a dummy Bitmap, to set in case the HTTP request generates an exception (404 and similar).

Imports System.Collections.Generic
Imports System.Drawing
Imports System.Linq
Imports System.Net
Imports System.Net.Http
Imports System.Security.Authentication
Imports System.Threading.Tasks

Public Class DownloadImagesHelper
    Implements IDisposable

    Private Shared ReadOnly client As New Lazy(Of HttpClient)(
        Function()
            Dim handler As New HttpClientHandler() With {
                .SslProtocols = SslProtocols.Tls12,
                .AllowAutoRedirect = True,
                .AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
                .CookieContainer = New CookieContainer()
            }
            Dim client As New HttpClient(handler)
            client.DefaultRequestHeaders.Add("Cache-Control", "no-cache")
            client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate")
            client.DefaultRequestHeaders.ConnectionClose = False
            Return client
        End Function)

    Public Sub New()
    End Sub

    Public Property DummyImage As Image = Nothing

    Public Async Function DownloadImages(urlList As IEnumerable(Of String)) As Task(Of List(Of Image))
        Dim tasks = urlList.Select(Function(url, idx) GetImage(idx, New Uri(url)))
        Await Task.WhenAll(tasks).ConfigureAwait(False)
        Return tasks.OrderBy(Function(t) t.Result.Key).Select(Function(t) t.Result.Value).ToList()
        ' Or, depending what you have decided to do in GetImage()
        ' only return results that have a non-null Image
        ' Return tasks.OrderBy(Function(t) t.Result.Key).Where(Function(t) t.Result.Value IsNot Nothing).Select(Function(t) t.Result.Value).ToList()
    End Function

    Private Async Function GetImage(pos As Integer, url As Uri) As Task(Of KeyValuePair(Of Integer, Image))
        Dim imageData As Byte() = Nothing
        Try
            imageData = Await client.Value.GetByteArrayAsync(url).ConfigureAwait(False)
            Return New KeyValuePair(Of Integer, Image)(
            pos, DirectCast(New ImageConverter().ConvertFrom(imageData), Image)
        )
        Catch hrEx As HttpRequestException
            ' Or return a null Image: Return New KeyValuePair(Of Integer, Image)(pos, Nothing)
            Return New KeyValuePair(Of Integer, Image)(pos, DummyImage)
        End Try
    End Function

    Public Sub Dispose() Implements IDisposable.Dispose
        Dispose(True)
        GC.SuppressFinalize(Me)
    End Sub
    Protected Overridable Sub Dispose(disposing As Boolean)
        client?.Value?.CancelPendingRequests()
        client?.Value?.Dispose()
    End Sub
End Class