Issue with connecting AWS IoT Rule to a MSK Kafka instance in a separate AWS account-CodePudding

Thank you in advance for any help!

My company uses multiple AWS accounts to separate region and "global" functionality/storage. We've recently switched to using MQTT IoT Rules and have run tests with regional Kafka vs a global Kafka instance for how we process data, deciding that a global Kafka makes the most sense.

I've been able to properly setup a regional Kafka with the regional IoT pushing data to it (in the same account), but when I tried setting up a global Kafka with the regional IoT pushing data it failed (separate accounts).

The IoT Rule also does not post any errors to Cloudwatch Logs, despite setting up a log group as an error action.

We've set everything up to use peer connections and allowing CIDR access to security groups and have many other services running from region to global just fine, but this one is failing us.

I've even been able to create an EC2 instance in the region under the same VPC, Subnet, and Security group as the ones I've assigned to my IoT Rule and it's working just fine to produce and consume from Kafka.

The only place I can imagine the issue being is with the bootstrap URLs not being accessible inside the region or the secret username/password not having proper access. The secret has been set in the global account. The role that I'm using has been given permission to read that secret in the separate account.

arn:aws:iam::12345:role/TestRole:

{
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "arn:aws:secretsmanager:region:9876:secret:AmazonMSK_TestSecret"
        }

${get_secret('arn:aws:secretsmanager:region:9876:secret:AmazonMSK_TestSecret', 'SecretString', 'username', 'arn:aws:iam::12345:role/TestRole')}

If anyone is familiar with reading secrets across accounts or using MSK Bootstrap URLs across accounts I'd appreciate some insight.

CodePudding user response：

AWS IoT rules supports cross account but only for a limited list of services: Amazon SQS, Amazon SNS, Amazon S3, and AWS Lambda.

You can see this information here.

So in your case, you would have to push data into a SQS queue in the other account and then pull data using Lambda in order to ingest it to MSK.