Total progress for multiple async uploads via BlobClient.UploadAsync()-CodePudding

I tested and found that for uploading a folder full of files to an Azure Blob Storage container, the fastest method (with the v12 Azure Storage SDK) involves using a Queue (Queue<Task<Response<BlobContentInfo>>>) and firing off an UploadAsync() for each file, ending in a Task.WhenAll(). The blob client coordinates max concurrency, etc. as configured. However, I need to show total progress across all files, and when I use the same ProgressHandler for each blob client, the Progress<long>() event only returns total bytes uploaded per file, without any context for the underlying file whose progress is being reported. How can I get the cumulative progress of multiple async uploads?

using Azure;
using Azure.Storage;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;

public async Task UploadBlobs(string sourcePath)
{
  var blobContainerClient = new BlobContainerClient(connectionString, containerName);
  var tasks = new Queue<Task<Response<BlobContentInfo>>>();
  var progressHandler = new Progress<long>();
  progressHandler.ProgressChanged  = UploadProgressChanged;

  var options = new BlobUploadOptions()
  {
      ProgressHandler = progressHandler,
      TransferOptions = new StorageTransferOptions()
      {
          MaximumConcurrency = Environment.ProcessorCount * 2,
          MaximumTransferSize = 50 * 1024 * 1024
      }
  };

  foreach (string filePath in Directory.GetFiles(sourcePath))
  {
      string fileName = Path.GetFileName(filePath);
      Console.WriteLine($"Uploading {fileName}\r\n");
      var blobClient = blobContainerClient.GetBlobClient(fileName);

      tasks.Enqueue(blobClient.UploadAsync(filePath, options));
  }

  await Task.WhenAll(tasks);
}

private void UploadProgressChanged(object sender, long bytesUploaded)
{
    // This handles every progress change for every file, but with no file context info!
}

CodePudding user response：

One possible solution to track individual blob's upload progress is to create your own progress handler and pass BlobClient to it.

Here's a rudimentary implementation of this:

public class BlobUploadProgressChange : Progress<long>
{
    private readonly BlobClient _blobClient;

    public BlobUploadProgressChange(BlobClient blobClient) : base()
    {
        _blobClient = blobClient;
    }

    public BlobClient BlobClient
    {
        get { return _blobClient; }
    }
}

This is how you would modify the UploadProgressChanged event handler:


void UploadProgressChanged(object? sender, long bytesUploaded)
{
    BlobUploadProgressChange item = (BlobUploadProgressChange) sender;
    Console.WriteLine(item.BlobClient.Name);
    Console.WriteLine($"Bytes uploaded: {bytesUploaded}");
    Console.WriteLine("====================================");
}

and this is how you can use it:

foreach (string filePath in Directory.GetFiles(sourcePath))
{
    string fileName = Path.GetFileName(filePath);
    Console.WriteLine($"Uploading {fileName}\r\n");
    var blobClient = blobContainerClient.GetBlobClient(fileName);
    var progressHandler = new BlobUploadProgressChange(blobClient);
    progressHandler.ProgressChanged  = UploadProgressChanged;
    var options = new BlobUploadOptions()
    {
        ProgressHandler = progressHandler
    };
    tasks.Enqueue(blobClient.UploadAsync(filePath, options));
}