I am using grafana loki stack in EKS. I am trying to access cloudwatch metrics using IRSA.

Pasted below is the IAM Policy and trust relations ship json for the role named "prometheus-monitoring-storage"

Still I am getting the error:

metric request error: "AccessDenied: User: arn:aws:sts::999999999999:assumed-role/data-services-test-monitoring-role/2759368202030603915 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::999999999999:role/data-services-test-monitoring-role\n\tstatus code: 403, request id: 3e72898d-98c4-43b5-b075-033f384581c3"

Please take a look at the below JSONs and help me figure out where am I committing a mistake.

ROLE POLICY

{
    "Statement": [
        {
            "Action": [
                "tag:GetResources",
                "s3:Put*",
                "s3:ListBucket",
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation",
                "s3:Get*",
                "s3:Delete*",
                "logs:StopQuery",
                "logs:StartQuery",
                "logs:GetQueryResults",
                "logs:GetLogGroupFields",
                "logs:GetLogEvents",
                "logs:DescribeLogGroups",
                "ec2:DescribeTags",
                "ec2:DescribeRegions",
                "ec2:DescribeInstances",
                "dynamodb:UpdateTable",
                "dynamodb:UpdateItem",
                "dynamodb:UntagResource",
                "dynamodb:TagResource",
                "dynamodb:Query",
                "dynamodb:PutItem",
                "dynamodb:ListTagsOfResource",
                "dynamodb:ListTables",
                "dynamodb:GetItem",
                "dynamodb:DescribeTable",
                "dynamodb:DeleteTable",
                "dynamodb:DeleteItem",
                "dynamodb:CreateTable",
                "dynamodb:BatchWriteItem",
                "dynamodb:BatchGetItem",
                "cloudwatch:ListMetrics",
                "cloudwatch:GetMetricData",
                "cloudwatch:GetInsightRuleReport",
                "cloudwatch:DescribeAlarmsForMetric",
                "cloudwatch:DescribeAlarms",
                "cloudwatch:DescribeAlarmHistory"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::data-services-test-thanos/*",
                "arn:aws:s3:::data-services-test-thanos",
                "arn:aws:s3:::data-services-test-loki/*",
                "arn:aws:s3:::data-services-test-loki",
                "arn:aws:logs:ap-southeast-1:165243018154:*:*",
                "arn:aws:dynamodb:ap-southeast-1:999999999999:table/data-services-test-loki/index/*",
                "arn:aws:dynamodb:ap-southeast-1:999999999999table/data-services-test-loki",
                "arn:aws:cloudwatch:ap-southeast-1:999999999999:*:*"
            ],
            "Sid": "MonitoringPolicy"
        }
    ],
    "Version": "2012-10-17"
}

TRUST RELATIONSHIP
------------------
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::999999999999:oidc-provider/oidc.eks.ap-southeast-1.amazonaws.com/id/A1B2C3D4E5F6E7F8G9H1J2L3M4N5P6
            },
            "Action": [
                "sts:AssumeRoleWithWebIdentity"
            ],
            "Condition": {
                "StringEquals": {
                    "oidc.eks.ap-southeast-1.amazonaws.com/id/A1B2C3D4E5F6E7F8G9H1J2L3M4N5P6:sub": "system:serviceaccount:monitoring:prometheus-monitoring-storage"
                }
            }
        }
    ]
}

CodePudding user response：

You didn't provide reproducible example, so this is only guess. Allow to assume role in AssumeRolePolicyDocument section, e.g.:

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Description": "AWS CloudFormation to create roles for Grafana cross account access",
    "Resources": {
        "GrafanaRole": {
            "Type": "AWS::IAM::Role",
            "Properties": {
                "RoleName": "grafana-cross-account-role",
                "AssumeRolePolicyDocument": {
                    "Version": "2012-10-17",
                    "Statement": [{
                        "Effect": "Allow",
                        "Principal": {
                            "AWS": "<TODO>"
                        },
                        "Action": ["sts:AssumeRole"]
                    }]
                },
                "Policies": [{
                    "PolicyName": "GrafanaRole",
                    "PolicyDocument": {
                        "Version": "2012-10-17",
                        "Statement": [{
                                "Sid": "AllowReadingMetricsFromCloudWatch",
                                "Effect": "Allow",
                                "Action": [
                                    "cloudwatch:DescribeAlarmsForMetric",
                                    "cloudwatch:DescribeAlarmHistory",
                                    "cloudwatch:DescribeAlarms",
                                    "cloudwatch:ListMetrics",
                                    "cloudwatch:GetMetricStatistics",
                                    "cloudwatch:GetMetricData"
                                ],
                                "Resource": "*"
                            },
                            {
                                "Sid": "AllowReadingLogsFromCloudWatch",
                                "Effect": "Allow",
                                "Action": [
                                    "logs:DescribeLogGroups",
                                    "logs:GetLogGroupFields",
                                    "logs:StartQuery",
                                    "logs:StopQuery",
                                    "logs:GetQueryResults",
                                    "logs:GetLogEvents"
                                ],
                                "Resource": "*"
                            },
                           {
                                "Sid": "AllowReadingTagsInstancesRegionsFromEC2",
                                "Effect": "Allow",
                                "Action": [
                                    "ec2:DescribeTags",
                                    "ec2:DescribeInstances",
                                    "ec2:DescribeRegions"
                                ],
                                "Resource": "*"
                            },
                            {
                                "Sid": "AllowReadingResourcesForTags",
                                "Effect": "Allow",
                                "Action": "tag:GetResources",
                                "Resource": "*"
                            }
                        ]
                    }
                }]
            }
        }
    }
}

Replace <TODO> based on your need, e.g. arn:aws:sts::999999999999:root if you want to allow access from all resources from that particular AWS account id 999999999999. It can be also role, maybe user, ....

CodePudding user response：

Thanks a lot for the help. I added the content shown in Policies[] array and it solved my problem. I am creating IAM Roles and Policies using Terraform script. Once again thank you very much for the hint.