I need to copy a folder within the same bucket and then run various logic on the copied content. I wanted to know if there's a way to copy the entire folder without using listObjects and then proceeding to copying each file separately.

Because this would mean running listObjects and copying each file and then doing listObjects again on the new folder and then running logic on each file.

So basically I'm trying to save IO and avoid multiple loops.

Please advise.

CodePudding user response：

You can use the --recursive tag in your SDK of choice to accomplish this. Combine this with the --include and --exclude flags which can use wild cards, you can achieve your goal. See this page of the CLI documentation

something like:

aws s3 cp s3://mybucket/logs/ s3://mybucket/logs2/ --recursive --exclude "*" --include "*.log"

CodePudding user response：

Amazon S3 does not provide a command that 'copies a folder'. Instead, each object must be individually copied via its own API request.

This means that you will first need to obtain a listing of the objects. This can be obtained via:

• A call to ListObjects (note: It can only return a maximum of 1000 objects per API call)

OR

• Use Amazon S3 Inventory to generate a list of existing objects in CSV format, and then use that list to generate the Copy requests

If you have a large number of objects, you could consider using Amazon S3 Batch Operations, which can copy files or invoke an AWS Lambda function for each object.

You could also configure Amazon S3 to trigger an AWS Lambda function whenever an object is Created (including when it is copied). Thus, the creation of the object can directly trigger the logic that you want to run.