Download a list of specific files from Azure Blob-CodePudding

I've got an issue with the files download from Azure Blob container. This is not that trivial as it may seem. I saw many examples of how you can download one file, but there's a problem if you need to download a bulk of them.

Issue definition:

I have an Azure Blob container that has ~30k files in it
At the same time, I've got a list of exact file names locally (around 300 files) that I want to download from that Azure Blob container (i.e., I need to download a whole bunch of files selectively, by their names)

I know that all these files exist in the given blob. So, I need a way to iterate over the list of the files and download them from the blob.

What I tried:

I tried 'azcopy copy' command. It works alright if you need to copy one or several files from the blob to your local disk, but there's no way you can pass a huge list of files as a parameter to download those
Tried to search for PowerShell examples that can do the same, but no luck

Please advice.

CodePudding user response：

Please try something like the following. It makes use of Get-AzStorageBlobContent Cmdlet.

The idea is to have an array of blobs you wish to download and then loop over that array and call this Cmdlet for each item.

$accountName = "account-name"
$accountKey = "account-key"
$containerName = "container-name"
$context = New-AzStorageContext -StorageAccountName $accountName -StorageAccountKey $accountKey
$destination = "C:\temp"

$blobNames = @("blob1.txt", "blob2.txt", "blob3.txt", "blob4.txt")

For ($i=0; $i -lt $blobNames.Length; $i  ) {
    $blob = $blobNames[$i]
    Write-Host "Downloading $blob. Please wait."
    Get-AzStorageBlobContent -Blob $blob -Container $containerName -Destination $destination -Context $context -Verbose
}

CodePudding user response：

Have you tried "Azure Storage Explorer" soft ? I was able to download a whole folder from a blob storage thanks to it.

If data in the blob container is in a folder just right click on the folder > Download.

If the files are directly at the root of the container (not stored in a subfolder) you'll have to select all files with the "Select All > Select all files in cache" option then click on "Download".

CodePudding user response：

$accountName = "account-name"
$accountKey = "account-key"
$containerName = "container-name"
$context = New-AzStorageContext -StorageAccountName $accountName -StorageAccountKey $accountKey
$destination = "C:\temp"

$blobNames = Get-Content $destination\list.txt

For ($i=0; $i -lt $blobNames.Length; $i  ) {
    $blob = $blobNames[$i]
    Write-Host "Downloading $blob. Please wait."
    Get-AzStorageBlobContent -Blob $blob -Container $containerName -Destination $destination -Context $context -Verbose
}

It'll work like a charm, given that the text file that holds a list of file names for the blobs you need to download is located in the '$destination' directory (could be any directory on your PC, though). p.s., the files just need to be listed as one column (separated by a caret return, i.e., "\n" at the end of each file name). Thanks @Gaurav Mantri for the solution.