Amazon S3 Copy files after date and with regex-CodePudding

I'm trying to copy some files from S3 sourceBucket to targetBucket, but I need to filter by date and by prefix. I wish it could be done with AWS CLI, but at the moment I'm stuck with list-object or with cp command.

I can filter correctly with

aws s3api list-objects-v2 --bucket sourceBucket --query 'Contents[?(LastModified > `2021-09-01`)]' --prefix "somePrefix_"

With the CP I can copy the files, but only by prefix

aws s3 cp s3://sourceBucket/ s3://targetBucket/ --recursive --include "somePrefix" --exclude "*"

I tried to come up with some ideas using the header --x-amz-copy-source-if-modified-since but it looks like you can use it with the command aws s3api copy-object and it copies one item at a time (doc). I read some answers/docs and I think I understood che cp command doesn't filter by date, but only by prefix.

Do you have any idea on how to solve this? Thank you in advance!

CodePudding user response：

Since you already have a list with objects you want to copy to another bucket, I suggest writing a bash script which does the copying for multiple objects:

#!/bin/bash

SOURCE_BUCKET="<my-bucket>"
DESTINATION_BUCKET="<my-other-bucket>"
PREFIX="<some-prefix>"

content=$(aws s3api list-objects-v2 --bucket $SOURCE_BUCKET --query 'Contents[?(LastModified > `2021-09-01`)]' --prefix $PREFIX | jq -r ".[].Key")

for file in $content;
do
    aws s3api copy-object --copy-source $SOURCE_BUCKET/$file --key $file --bucket $DESTINATION_BUCKET | jq
done

Please note, this scripts requires jq to be installed.