I've deployed an EKS cluster using the Terraform module terraform-aws-modules/eks/aws. I’ve deployed Airflow on this EKS cluster with Helm (the official chart, not the community one), and I’ve annotated worker pods with the following IRSA:

  serviceAccount:
    # Specifies whether a ServiceAccount should be created
    create: true
    # The name of the ServiceAccount to use.
    # If not set and create is true, a name is generated using the release name
    name: "airflow-worker"

    # Annotations to add to worker kubernetes service account.
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::123456789:role/airflow-worker"

This airflow-worker role has a policy attached to it to enable it to assume a different role.

I have a Python program that assumes this other role and performs some S3 operations. I can exec into a running BashOperator pod, open a Python shell, assume this role, and issue the exact same S3 operations successfully.

But, when I create a Docker image with this program and try to call it from a KubernetesPodOperator task, I see the following error:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the AssumeRole operation: 
User: arn:aws:sts::123456789:assumed-role/core_node_group-eks-node-group-20220726041042973200000001/i-089c64b96cf7878d8 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::987654321:role/TheOtherRole

I don't really know what this role is, but I believe it was created automatically by the Terraform module. However, when I kubectl describe one of these failed pods, I see this:

    Environment:
      ...
      ...
      ...
      AWS_ROLE_ARN: arn:aws:iam::123456789:role/airflow-worker

My questions:

1. Why is this role being used, and not the IRSA airflow-worker that I've specified in the Helm chart's values?
2. What even is this role? It seems the Terraform module creates a number of roles automatically, but it is very difficult to tell what their purpose is or where they're used from the Terraform documentation.
3. How am I able to assume this role and do everything the Dockerized Python program does when in a shell in the pod? Okay, this is because other operators (such as BashOperator) do use the airflow-worker role. Just not KubernetesPodOperators.
4. What is the AWS_IAM_ROLE environment variable, and why isn't it being used?

Happy to provide more context if it's helpful.

CodePudding user response：

A lot my questions remain unanswered, but I fixed the ultimate issue by passing service_account="airflow-worker" to KubernetesPodOperator(....

CodePudding user response：

In order to use the AWS role in EKS pod, you need to add this policy to it:

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "AWS": " arn:aws:iam::123456789:role/airflow-worker” 
        },
        "Action": "sts:AssumeRole"
      }
    ]
  }

Here you can find some information about AWS Security Token Service (STS).

For the tasks running in the worker prod, they will use the role automatically, but if you create a new pod, it will be separated from your worker pod, so you need to let it use the service account which attach the role in order to add the AWS role creds file to the pod.