I have an issue to which I cannot find the right documentation to solve it or get any good path to follow.

I have several azure storage containers which have multiple containers and blobs.

I am trying to loop through each storage account and have a list of its containers, so I can copy them into another storage account as a backup.

Yesterday with the help of the community, I was able to make work the copy from storage account to storage account as follow:

from azure.storage.blob import BlobClient, BlobServiceClient, ContainerClient
from azure.storage.blob import ResourceTypes, AccountSasPermissions
from azure.storage.blob import generate_account_sas    
from datetime import *




#================================ SOURCE ===============================
# Source Client
connection_string = '' # The connection string for the source container
account_key = '' # The account key for the source container
source_container_name = '' # Name of container which has blob to be copied
blob_name = '' # Name of the blob you want to copy




# Create client
client = BlobServiceClient.from_connection_string(connection_string) 



# Create sas token for blob
sas_token = generate_account_sas(
    account_name = client.account_name,
    account_key = account_key, 
    resource_types = ResourceTypes(object=True, container=True),
    permission= AccountSasPermissions(read=True,list=True),
    # start = datetime.now(),
    expiry = datetime.utcnow()   timedelta(hours=4) # Token valid for 4 hours
)

# Create blob client for source blob
source_blob = BlobClient(
    client.url,
    container_name = source_container_name, 
    blob_name = blob_name,
    credential = sas_token
)


# ============================= TARGET =======================================

# Target Client
target_connection_string = ''
target_account_key = ''
source_container_name = source_container_name
target_blob_name = ''
target_destination_blob = ''

# Create target client
target_client = BlobServiceClient.from_connection_string(target_connection_string)
container = ContainerClient.from_connection_string(target_connection_string, target_destination_blob)

# Create new blob and start copy operation.
# new_blob = client.get_blob_client(destination_container_name, blob_name)    
new_blob = target_client.get_blob_client(target_destination_blob, target_blob_name)
new_blob.start_copy_from_url(source_blob.url)

This Block of code works just fine, but as you can see, I have to hardcode the container and the blob name.

What I am trying to do, is iterate through all containers in a specific storage account, and start copying the content into the same location in another storage account.

So far the only documentation I found on GitHub is related to this:

https://github.com/Azure/azure-storage-python/issues/389

But I cannot get over that problem.

I managed to achieve this kind of loop over storage account but related to tables, by using the library from azure.cosmosdb.table.tableservice import TableService,ListGenerator

So I was wondering is somebody can help me to understand what library there is around to help to achieve this point. Thank you very much in advance for any help you can provide me with.

UPDATE:

all_containers = client.list_containers(include_metadata=True)

for container in all_containers:
    print(container['name'], container['metadata'])
    
container_client = BlobServiceClient.get_container_client(container['name'])
    

blobs_list = container_client.list_blobs()
for blob in blobs_list:
    print(blob.name   '\n')

CodePudding user response：

You would want to use Azure Storage Blobs client library for Python.

To list the containers, you will need to use list_containers method on your blob service client object.

   all_containers = client.list_containers(include_metadata=True)
   for container in all_containers:
       print(container['name'], container['metadata'])

Then using the container['name'], you will create an instance of ContainerClient. Once you have that, all you need to do is call list_blobs method to list the blobs in that container.

   blobs_list = container_client.list_blobs()
   for blob in blobs_list:
       print(blob.name   '\n')

UPDATE

Please try the code below. It will list all containers and blobs inside them in a storage account and print both container name and blob name.

from azure.storage.blob import BlobServiceClient

connection_string = "DefaultEndpointsProtocol=https;AccountName=accountname;AccountKey=accountkey"
client = BlobServiceClient.from_connection_string(connection_string)
all_containers = client.list_containers(include_metadata=True)
for container in all_containers:
    print(container['name'], container['metadata'])
    print("==========================")
    container_client = client.get_container_client(container.name)
    print(container_client)
    blobs_list = container_client.list_blobs()
    for blob in blobs_list:
        print(blob.name)
        print("==========================")