To transfer only objects greater than 100MB to AWS Glacier from S3 Standard using AWS Lifecycle Mana-CodePudding

I've 50TB of data in S3 Standard bucket. I want to push objects that are greater that 100MB as well as 30 days old to AWS Glacier using Lifecycle Management Policy. Can you let me know how to move only objects that are greater than 100MB in size. Thanks

CodePudding user response：

create a lambda that runs every day (cron job) that checks for files older than 30 days and greater then 100mb in the bucket. You can use the s3 api and glacier api.

CodePudding user response：

There is no way to transition items based on file size.

As the name suggests, S3 Lifecycle policies allow you to specify transition actions based on object lifetime - not file size - to move items from the S3 Standard storage class to S3 Glacier.

Now, a really inefficient & costly way that may be suggested would be to schedule a Lambda to check the S3 bucket daily, see if anything is 30 days old & then "move" items to Glacier.

However, the Glacier API does not allow you to move items from S3 Standard to Glacier unless it is through a lifecycle policy.

This means you will need to download the S3 object and then re-upload the item again to Glacier.

I would still advise having a Lambda running daily to check the file size of items, however, create another folder (key) called archive for example. If there are any items older than 30 days & greater than 100MB, copy the item from the current folder to the archive folder and then delete the original item.

Set a 0-day life-cycle policy, filtered on the prefix of the other folder (archive), which then transitions the items to Glacier ASAP.

This way, you will be able to transfer items larger than 100MB after 30 days, without paying higher per-request charges associated with uploading items to Glacier, which may even cost you more than the savings you were aiming for in the first place.

To later transition the object(s) back from Glacier to S3 Standard, use the RestoreObject API (or SDK equivalent) to restore it back into the original folder. Then finally, delete the object from Glacier using a DELETE request to the archive URL.