How to sync 2 s3 buckets, every day, with 2 different AWS accounts-CodePudding

I have AWS account ACCOUNT-A and a bucket under it BUCKET-DESTINATION. I have another AWS account ACCOUNT-B, with no access to it, but have been provided with a IAM user IAM-B and a bucket BUCKET-SOURCE that IAM-B has access to (Read/Write).

I want to use IAM-B to sync BUCKET-SOURCE to BUCKET-DESTINATION, every night. If possible all within AWS, I don't want to cron aws sync.

How would I go about doing this?

Thank you!

CodePudding user response：

You could set up S3 bucket replication across the two different accounts. The replication works asynchronously (usually within 15min, but sometimes up to a couple of hours as described in How long does object replication take on Amazon S3?). No additional services are needed to trigger the replication.

You will need to give account A permissions on bucket B (target bucket) via a bucket policy in order for the replication to work. The detailed configuration description can be found in Configuring replication when source and destination buckets are owned by different accounts.

CodePudding user response：

You will need to have a single set of credentials that has:

Read access to the source bucket, and
Write access to the destination bucket

Since you have been provided with with IAM User B that has access to the source bucket, and presuming that the destination bucket is under your control, the easiest method would be to create a bucket policy on the destination bucket that grants write access (PutObject) to that IAM User B.

This way, you can use a single set of credentials that has permission to read from the source bucket and write to the destination bucket.

Important: When writing to a bucket with a set of credentials that does not belong to the same account as the destination bucket, it is important to specify --acl bucket-owner-full-control. This sets the 'owner' of the object to the destination account.

It is not possible to schedule an AWS CLI sync operation within S3, so your choices are:

Use a cron job on a computer somewhere on the Internet, or
Schedule an AWS Lambda function that will copy the objects, using hard-coded credentials to operate as IAM User B rather than as the IAM Role assigned to the function, or
Have Account-B configure S3 Replication so that S3 immediately replicates objects between the buckets (without having to call a Copy command)

The last option is by far the easiest! (It would require a bucket policy on the destination bucket that permits the replication process to write objects to the bucket.)