Shell script to fetch S3 bucket size with AWS CLI-CodePudding

I have this script that fetches all the buckets in AWS along with the size. But when I am running the script its fetching the bucket, but when running the loop for fetching the size, its throwing error. can someone point me where I am going wrong here. bcos when I am running the awscli commands for individual bucket, its fetching the size without any issues. The desired output wille be as below, but for all the buckets, I have fetched for one bucket. Desired ouptut:

aws --profile aws-stage s3 ls s3://<bucket> --recursive --human-readable --summarize | awk END'{print}'
   Total Size: 75.1 KiB

Error:

 Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9] :[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"

Script:

#!/bin/bash
aws_profile=('aws-stage' 'aws-prod');

#loop AWS profiles
for i in "${aws_profile[@]}"; do
    echo "${i}"
    buckets=$(aws --profile "${i}" s3 ls s3:// --recursive | awk '{print $3}')

    #loop S3 buckets
    for j in "${buckets[@]}"; do
        echo "${j}"
        aws --profile "${i}" s3 ls s3://"${j}" --recursive --human-readable --summarize | awk END'{print}'
    done

done

CodePudding user response：

Try this:

#!/bin/bash

aws_profiles=('aws-stage' 'aws-prod');

for profile in "${aws_profiles[@]}"; do
    echo "$profile"
    read -rd "\n" -a buckets <<< "$(aws --profile "$profile" s3 ls | cut -d " " -f3)"
    for bucket in "${buckets[@]}"; do
        echo "$bucket"
        aws --profile "$profile" s3 ls s3://"$bucket" --human-readable --summarize | awk END'{print}'
    done
done

The problem was that your buckets was a single string, rather than an array.