Retrieve list of buckets from all regions using boto3-CodePudding

I was trying to retrieve the list of buckets from all regions using boto3, however I am unable to list the buckets from the correct region

So far I have tried location['LocationConstraint'], which comes up as None I have also tried the following as well but it didn't work.

Any help is appreciated, thank you

if client.head_bucket(Bucket=bucket['Name'])['ResponseMetadata']['HTTPHeaders']['x-amz-bucket-region'] == 'us-east-1':
                print ("bucketname %s " % s3_bucket.name)

Code

import json
import boto3

bucketlist = []

def lambda_handler(event, context):

    # Get list of regions
    ec2 = boto3.client('ec2')
    regions = ec2.describe_regions().get('Regions',[] )

    # Iterate over regions
    for region in regions:
    
        print ("*************** Checking region  --   %s " % region['RegionName'])
        reg=region['RegionName']
    
        client = boto3.client('s3', region_name=reg)

        response = client.list_buckets()
        for bucket in response['Buckets']:
            s3 = boto3.resource('s3', region_name=reg)
            s3_bucket = s3.Bucket(bucket['Name'])
            if client.head_bucket(Bucket=bucket['Name'])['ResponseMetadata']['HTTPHeaders']['x-amz-bucket-region'] == 'us-east-1':
                print ("bucketname %s " % s3_bucket.name)
                bucketlist.append(s3_bucket)
    return {
        "statusCode": 200
    }

Output: The buckets testbucket1 and testbucket2 are only available in us-east-1 region

START RequestId: e22f6ac0-7bb9-4e2b-84d7-5512ce97acfa Version: $LATEST
*************** Checking region  --   eu-north-1 
bucketname testbucket1 
bucketname testbucket2 
*************** Checking region  --   ap-south-1 
bucketname testbucket1 
bucketname testbucket2 
[...]
*************** Checking region  --   us-east-1 
bucketname testbucket1
bucketname testbucket2

Expected Output:

START RequestId: e22f6ac0-7bb9-4e2b-84d7-5512ce97acfa Version: $LATEST
*************** Checking region  --   eu-north-1 
*************** Checking region  --   ap-south-1 
*************** Checking region  --   eu-west-3 
*************** Checking region  --   eu-west-2 
*************** Checking region  --   eu-west-1 
*************** Checking region  --   ap-northeast-3 
*************** Checking region  --   ap-northeast-2 
*************** Checking region  --   ap-northeast-1 
*************** Checking region  --   sa-east-1 
*************** Checking region  --   ca-central-1 
*************** Checking region  --   ap-southeast-1 
*************** Checking region  --   ap-southeast-2 
*************** Checking region  --   eu-central-1 
*************** Checking region  --   us-east-1 
bucketname testbucket1 
bucketname testbucket2 
*************** Checking region  --   us-east-2 
*************** Checking region  --   us-west-1 
*************** Checking region  --   us-west-2

CodePudding user response：

You don't have to change region for client. list_buckets returns all buckets in the account, regardless where they are located. That's why you get the same buckets for each region in your code.

You have to iterate over list_buckets results, and use get_bucket_location to obtain the actual locations of your buckets. For example:

all_buckets = client.list_buckets()

for bucket in all_buckets['Buckets']:
  bucket_bame = bucket["Name"]
  region = client.get_bucket_location(Bucket=bucket_bame)["LocationConstraint"]
  print(bucket_bame, region)