I'm trying to build a firehose that delivers data to Opensearch using CDK with Python, when I do that from Console it works like a charm however when I try to deploy it with CDK it gives me this error

Resource handler returned message: "Verify that the IAM role has access to the ElasticSearch domain.

It is deriving me crazy, I tried every IAM es policy but no one works. this is my configuration here I define the role

        self.firehose_role = Role(
            self,
            "FirehoseRole",
            role_name=f"{construct_name}",
            assumed_by=ServicePrincipal(service="firehose.amazonaws.com"),
        )

        self.firehose_role.add_to_policy(
            PolicyStatement(
                actions=["es:*"],
                resources=[
                    "arn:aws:es:eu-west-1:xxx:domain/my-domain",
                    "arn:aws:es:eu-west-1:xxx:domain/my-domain/*",
                ],
            ),
        )

        self.firehose_role.add_to_policy(
            PolicyStatement(
                actions=[
                    "ec2:DescribeVpcs",
                    "ec2:DescribeVpcAttribute",
                    "ec2:DescribeSubnets",
                    "ec2:DescribeSecurityGroups",
                    "ec2:DescribeNetworkInterfaces",
                    "ec2:CreateNetworkInterface",
                    "ec2:CreateNetworkInterfacePermission",
                    "ec2:DeleteNetworkInterface",
                ],
                resources=[
                    "*",
                ],
            ),
        )

Then setting OS configuration as following

        os_config =CfnDeliveryStream.AmazonopensearchserviceDestinationConfigurationProperty(
            index_name="xxx",
            role_arn=self.firehose_role.role_arn,
            s3_configuration=CfnDeliveryStream.S3DestinationConfigurationProperty(
                #bucket config
            ),
            buffering_hints=CfnDeliveryStream.ElasticsearchBufferingHintsProperty(
                interval_in_seconds=120,
                size_in_m_bs=5,
            ),
            cloud_watch_logging_options=CfnDeliveryStream.CloudWatchLoggingOptionsProperty(
                enabled=True,
                log_group_name=log_group.log_group_name,
                log_stream_name=log_stream.log_stream_name,
            ),
            domain_arn="arn:aws:es:eu-west-1:xxx:domain/my-domain",
            retry_options=CfnDeliveryStream.AmazonopensearchserviceRetryOptionsProperty(
                duration_in_seconds=180
            ),
            s3_backup_mode="AllDocuments",
            vpc_configuration=CfnDeliveryStream.VpcConfigurationProperty(
                role_arn=self.firehose_role.role_arn,
                security_group_ids=["xxx"],
                subnet_ids=["xxx", "xxx"],
            ),
        )

To finally setting Kinesis Delivery stream

        self.delivery_stream = CfnDeliveryStream(
            self,
            id="FirehoseDS",
            delivery_stream_name=f"{construct_name}-ds",
            delivery_stream_type="DirectPut",
            amazonopensearchservice_destination_configuration=os_config,
        )

I run out of ideas, PS: knowing that OS access policy is wide open and I don't think the problem is there because if it was there at least I would've been able to deploy Firehose the I would have problems with data not reaching, but now I'm not able to even deploy Firehose

CodePudding user response：

The access policy on your Opensearch domain could be preventing access to the IAM role depending on how its configured. This will also need to allow the IAM role used by firehose to perform the same actions on the domain. https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html

Assuming a custom policy has been set on the domain the following statement would be added into the access policy to give access to the role being used by firehose.

{
  "Sid": "AllowFirehose",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::xxx:role/firehose-role"
    ]
  },
  "Action": "es:*",
  "Resource": [
    "arn:aws:es:eu-west-1:xxx:domain/my-domain",
    "arn:aws:es:eu-west-1:xxx:domain/my-domain/*"
  ]
}

CodePudding user response：

If you enable fine-grained access control in opensearch, the IAM role ARN need to be added to opensearch through master user or use opensearch securiry dashboard.