AWS managed elasticsearch S3 snapshot permissions-CodePudding

I have an AWS managed Elasticsearch instance (older pre-"OpenSearch" variant). I want to enable snapshot backups to S3, but I can't seem to get the permissions correct.

I'm trying to use instance roles, not specific user credentials. I've tried following this guide as well as quite a few others that are very similar.

I have a role: arn:aws:iam::MYACCOUNTID:role/QA-elasticsearch-instance-role. That role has the following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowListS3BackupBucket",
            "Action": [
                "iam:PassRole",
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:iam::MYACCOUNTID:role/QA-elasticsearch-instance-role",
                "arn:aws:s3:::qa-mycompanyname-elasticsearch-backups"
            ]
        },
        {
            "Sid": "AllowBackupStorage",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::qa-mycompanyname-elasticsearch-backups/*"
            ]
        },
        {
            "Sid": "AllowElasticSearchPost",
            "Effect": "Allow",
            "Action": "es:ESHttpPost",
            "Resource": "arn:aws:es:us-east-1:MYACCOUNTID:domain/qa-elasticsearch"
        }
    ]
}

This role also has the following trust relationship:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

I have the following python script to try and create the snapshot repository:

import boto3
import requests
from requests_aws4auth import AWS4Auth

host = 'https://MYELASTICSEARCHENDPOINT.us-east-1.es.amazonaws.com/'
region = 'us-east-1' # For example, us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()

awsauth = AWS4Auth("MYACCESSKEY", "MYSECRETKEY", region, service)

# Register repository
path = '_snapshot/qa_snapshot_repository' # the Elasticsearch API endpoint
url = host   path

payload = {
    "type": "s3",
    "settings": {
        "bucket": "qa-MYCOMPANYNAME-elasticsearch-backups",
        "region": "us-east-1",
        "role_arn": "arn:aws:iam::MYACCOUNTID:role/QA-elasticsearch-instance-role"
    }
}

headers = {"Content-Type": "application/json"}

r = requests.put(url, auth=awsauth, json=payload, headers=headers)

print(r.status_code)
print(r.text)

When I run this, (python3 elasticsearch_test.py | jq .) I get the following:

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[qa_snapshot_repository] path  is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[qa_snapshot_repository] path  is not accessible on master node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [tests-4-VcUFXhTcW4PQ6fi6AiVA/master.dat] using a single upload",
      "caused_by": {
        "type": "amazon_s3_exception",
        "reason": "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: RJKPNBBEQQBNWHWT; S3 Extended Request ID: 3O5AqSkKhsCNlmYLCtcfnE0omhp6wdbPCejJ2r4dNJq93l2xppSs4Hv6uTGvNIJ2jFWLt31gsyc=)"
      }
    }
  },
  "status": 500
}

I've been googling and trying to tweak permissions for a few hours now, but can't seem to get it right. Is there something obvious I'm missing?

CodePudding user response：

So it turns out the error response was super misleading. It isn't really an "AccessDenied" problem, but rather the request to S3 is getting rejected because of improper encryption between client and server. Following this post I decided to try adding "server_side_encryption": "true", and everything worked.

payload = {
    "type": "s3",
    "settings": {
        "bucket": "qa-mycompanyname-elasticsearch-backups",
        "region": "us-east-1",
        "role_arn": "arn:aws:iam::myaccountid:role/QA-elasticsearch-instance-role",
        "server_side_encryption": "true"
    }
}

200
{
  "acknowledged": true
}

I have no proper explanation for why this worked. I guess it tells the client on the Elasticsearch instance to use https URLs to communicate with S3 instead of http, but seems nuts that http would be the default, or that there wouldn't be a better error message back.