I am trying to select one existing bucket then upload file into the bucket using Boto3. But all methods related with buckets aren't working correctly for me.

The problems I'm confronting are like this:

When I list up the buckets using boto3.Session.client('s3').list_buckets() method, it results an empty list of Buckets.
When I select a specific bucket using boto3.Session.resource('s3').Bucket('existing_bucket_name') then run upload_fileobj() method, it throws NoSuchBucket exception(which is very ridiculous the Bucket object's attribute name correctly returns the bucket's name though).

What I've tried so far:

used Session instance with keyword arguments indicating account and region configuration. I had some issues with Boto3 config setup and finally gave up using config file then directly gave the instance config arguments.
used resource instead of client. client method didn't work correctly, so I used resource but it also didn't go well.

here is my code:

Implemented client class

class S3Client:

    def __init__(self):
        region = 'ap-northeast-2'
        self.session = boto3.Session(
            aws_access_key_id=os.environ['AWS_ACCESS_KEY_ID'],
            aws_secret_access_key=os.environ['AWS_SECRET_ACCESS_KEY'],
            region_name=region
        )
        self.client = self.session.client(service_name='s3', region_name=region,
                                          aws_access_key_id=os.environ['AWS_ACCESS_KEY_ID'],
                                          aws_secret_access_key=os.environ['AWS_SECRET_ACCESS_KEY'])
        self.resource = self.session.resource('s3')
        self.bucket = self.resource.Bucket(settings.AWS_S3['BUCKET_NAME'])

    @staticmethod
    def open_test_file_as_byte():
        file_name = os.path.join(os.path.dirname(__file__), 'test_file.pdf')

        data = open(file_name, 'rb')
        return data

    def upload_file(self, file_name):
        with self.open_test_file_as_byte() as file_byte_stream:
            return self.bucket.upload_fileobj(
                file_byte_stream,
                'reports/'   file_name,
                ExtraArgs={'ACL': 'public-read-write'}
        )

    # this method worked once with message saying that creation successfully completed, but I cannot find anything in my S3
    def create_bucket(self):
        return self.client.create_bucket(
            ACL='public-read-write',
            Bucket='public-reports'
            # CreateBucketConfiguration={
            #     'LocationConstraint': 'ap-northeast-2'
            # }
        )

Codes that invoke the client's method

@mock_s3
class S3ClientTestCases(unittest.TestCase):
    @classmethod
    def setUpClass(cls) -> None:
        cls.s3_client = S3Client()

    def test_create_bucket(self):
        logger.debug(self.s3_client.create_bucket())
        logger.debug(self.s3_client.client.list_buckets())

    def test_upload_file(self):
        logger.debug(self.s3_client.upload_file('test.pdf'))

This is full error log:

======================================================================
ERROR: test_upload_file (apps.reports.test.S3ClientTestCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/psymon/Documents/Git/portfolio-manager/.venv/lib/python3.10/site-packages/moto/core/models.py", line 111, in wrapper
    result = func(*args, **kwargs)
  File "/Users/psymon/Documents/Git/portfolio-manager/apps/reports/test.py", line 54, in test_upload_file
    logger.debug(self.s3_client.upload_file('test.pdf'))
  File "/Users/psymon/Documents/Git/portfolio-manager/apps/reports/services/report_upload_service.py", line 71, in upload_file
    return self.bucket.upload_fileobj(
  File "/Users/psymon/Documents/Git/portfolio-manager/.venv/lib/python3.10/site-packages/boto3/s3/inject.py", line 678, in bucket_upload_fileobj
    return self.meta.client.upload_fileobj(
  File "/Users/psymon/Documents/Git/portfolio-manager/.venv/lib/python3.10/site-packages/boto3/s3/inject.py", line 636, in upload_fileobj
    return future.result()
  File "/Users/psymon/Documents/Git/portfolio-manager/.venv/lib/python3.10/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/Users/psymon/Documents/Git/portfolio-manager/.venv/lib/python3.10/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/Users/psymon/Documents/Git/portfolio-manager/.venv/lib/python3.10/site-packages/s3transfer/tasks.py", line 139, in __call__
    return self._execute_main(kwargs)
  File "/Users/psymon/Documents/Git/portfolio-manager/.venv/lib/python3.10/site-packages/s3transfer/tasks.py", line 162, in _execute_main
    return_value = self._main(**kwargs)
  File "/Users/psymon/Documents/Git/portfolio-manager/.venv/lib/python3.10/site-packages/s3transfer/upload.py", line 758, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/Users/psymon/Documents/Git/portfolio-manager/.venv/lib/python3.10/site-packages/botocore/client.py", line 515, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/psymon/Documents/Git/portfolio-manager/.venv/lib/python3.10/site-packages/botocore/client.py", line 934, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the PutObject operation: The specified bucket does not exist

EDIT #1

Bucket Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::some_bucket_name/*"
        }
    ]
}

Granted Permission to User:

AmazonS3FullAccess

I guess something is wrong with AWS account setting but I'm not sure what exactly cause this entire malfunctions...

Please any advice!

CodePudding user response：

Solved

The cause of my problem was the annotation on the test code, @mock_s3.

Actually, I cannot even call this as problem, but a mistake.

Anyway, if anybody get in trouble with that annotation, I hope my stupid record help you.