How can i download a local copy of an S3 snapshot of an AWS Postgres DB?-CodePudding

There is a snapshot in S3 of a Postgres DB, but the download button is grayed out... if i navigate to each file in each table, i am able to download the .gz.parquet files individually, but that is crazy.

So I installed the aws cli, configured a default user, tried to run aws s3 cp s3://<your-bucket-name>/<your-snapshot-name> <local-path> but always get:

fatal error: An error occurred (404) when calling the HeadObject operation: Key <your-snapshot-name> does not exist

But it does exist, and I can see it in the aws website and see the root folder if i run aws s3 ls.

So i tried aws s3 cp --recursive s3://<your-bucket-name>/<your-snapshot-name> <local-path> and it goes through all the folders, copies them to my computer, but theyre all empty folders, and i get the following error for every folder its going through:

An error occurred (AccessDenied) when calling the GetObject operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.

The permissions I'm using are a generic (what i thought was) all access to S3:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*",
                "s3-object-lambda:*"
            ],
            "Resource": "*"
        }
    ]
}

Plus two from here:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListAllMyBuckets"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::snapshots"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::snapshots/*"
            ]
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListAllMyBuckets"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::snapshots"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::snapshots/*"
            ]
        }
    ]
}

What am I missing here?

CodePudding user response：

The first error you are experiencing is probably because the aws s3 cp command works on objects, not directories. A good way to copy a whole directory (including subdirectories) is to use aws s3 sync.

The second error mentions "customer master key". This is probably referring to a KMS key that was used to encrypt the file when it was created by Amazon RDS. Try giving yourself kms:* permissions (although you probably only need kms:Decrypt) and it should be able to read the file.