I am trying to load csv files from one s3 bucket in one account to another. For accessing 2 accounts, I have written the following script

import boto3

source_session = boto3.Session(profile_name='account1')
source_s3 = source_session.client('s3')

destination_session = boto3.Session(profile_name='account2')
destination_s3 = destination_session.client('s3')

I saw that for upload_file() method, the parameters are the file location and the des_bucket name. How should i mention the file location for different accounts?
Something like this:

destination_s3.upload_file(source_file_location, des_bucket_name);

CodePudding user response：

I think that you need download the file from the source bucket first, so you can provide the disk file path to destination_s3.upload_file method.

CodePudding user response：

You can use the boto3 copy_object() command to tell S3 to copy an object between buckets. It will transfer totally within S3, without needing a download/upload.

However, given that the two buckets belong to different accounts, you will need to configure permissions so that the your program uses a single set of credentials that can both 'read' the source bucket and 'write' to the destination bucket (rather than using two different sets of credentials). You can do this in two ways:

Pull object

You can 'pull' an object by using a set of credentials associated with the destination bucket:

• Use credentials from an IAM User in the target account (User B)
• Grant permissions in IAM for User B to read from the target bucket
• Add a Bucket Policy to the Source bucket that permits User B to Read from the bucket

Push object

Alternatively, you can 'push' an object from the destination account to the target bucket. You would use a set of credentials associated with the source account:

• Use credentials from an IAM User in the source account (User A)
• Grant permissions in IAM to User A to read from the source bucket
• Add a Bucket Policy to the Destination bucket that permits User A to Write to the bucket
• When copying the object, specify ACL='bucket-owner-full-control', which will grant ownership of the object to the destination account. This is only required when the object is created using credentials from an account that does not own the bucket.