I'm trying to download a folder from a public AWS S3 bucket using the Python library cloudpathlib. My code looks like this:

from cloudpathlib import CloudPath
path = r"C:\some\path\to\folder"
url = "s3://some-example-bucket/folder/"

cloud_path = CloudPath(url)
cloud_path.download_to(path)

Really straight forward. To my knowledge, this should work, because the bucket is public:

Here is the bucket policy (nabbed from the AWS S3 tutorial):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicRead",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::cozy-auto-texture-sd-repo/*"
        }
    ]
}

When I run the Python code to install the folder stable-diffusion-v1-4 the following error message appears:

  File "E:\Cozy-Auto-Texture-Files\venv\lib\site-packages\botocore\auth.py", line 418, in add_auth
    raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials

My question is why is this happening? My bucket shouldn't require credentials since it's public. Is there something I'm missing with the buckets permissions or is it a Python code thing?

Thank you for reading and I appreciate the help!

Edit:

I've tried again with this method:

BUCKET_NAME = 'cozy-auto-texture-sd-repo'  # replace with your bucket name
KEY = 'stable-diffusion-v1-4'

s3 = boto3.resource('s3')

try:
    s3.Bucket(BUCKET_NAME).download_file(KEY, sd_path)
except botocore.exceptions.ClientError as e:
    if e.response['Error']['Code'] == "404":
        print("The object does not exist.")
    else:
        raise

However the same error message botocore.exceptions.NoCredentialsError: Unable to locate credentials appears, leading me to believe there is something wrong with my S3 bucket setup.

CodePudding user response：

Assuming you are running this script locally, you may need to first set up your AWS credentials on your local machine.

See this existing answer:

It worked with an ~/.aws/credentials file containing:

[default]
aws_access_key_id=XXXXXXXXXXXXXX
aws_secret_access_key=YYYYYYYYYYYYYYYYYYYYYYYYYYY

Using aws configure also works if you have aws-cli installed.

CodePudding user response：

It turns out that AWS S3 doesn't allow you to download a folder from a public bucket. I got around this by simply zipping the folder I was trying to distribute and unzipping it in my application when the download was complete.