Azure Powershell: How Do I search for files in a BLOB storage quickly?-CodePudding

We store log files in an Azure storage account, sorted in directories, by date and customer, like this:

YYYY/MM/DD/customerNo/.../.../somestring.customerNo.applicatoinID.log

I need to parse some of these files automatically every day which works fine. However, all I know is the prefix mentioned above and the suffix, they might be in different subdirectories.

So this is how I did it:

$files = (Get-AzStorageBlob -Container logfiles -Context $context) | Where-Object { $_.Name -like "$customerId.$appID.txt" }

This was fast while there weren't any log files, but now after a year this search takes ages. I read

From what I understand, Azure's BLOB storage does not have a hierarchical file system that supports folders, so the "/" are part of the BLOB name and are being interpreted as folders by client software.

However, that does not help me speeding up the search. Any suggestions on how to improve the situation?

CodePudding user response：

Azure Blob Storage supports server-side filtering of blobs by prefix however your code is not taking advantage of that.

$files = (Get-AzStorageBlob -Container logfiles -Context $context) | Where-Object { $_.Name -like "$prefix*$customerId.$appID.txt" }

Essentially the code above is listing all blobs and then doing the filtering on the client side.

To speed up the search, please modify your code to something like:

$files = (Get-AzStorageBlob -Container logfiles -Prefix $prefix -Context $context) | Where-Object { $_.Name -like "$prefix*$customerId.$appID.txt" }

I simply passed the prefix in the Prefix parameter. Now you'll receive only the blobs names of which start with the prefix .