Azure AKS/Container App can't access Key vault using managed identity-CodePudding

I have a docker container python app deployed on a kubernetes cluster on Azure (I also tried on a container app). I'm trying to connect this app to Azure key vault to fetch some secrets. I created a managed identity and assigned it to both but the python app always fails to find the managed identity to even attempt connecting to the key vault.

The Managed Identity role assignments:

Key Vault Contributor -> on the key vault

Managed Identity Operator -> Managed Identity

Azure Kubernetes Service Contributor Role, Azure Kubernetes Service Cluster User Role, Managed Identity Operator -> on the resource group that includes the cluster

Also on the key vault Access policies I added the Managed Identity and gave it access to all key, secrets, and certs permissions (for now)

Python code:

 credential = ManagedIdentityCredential()
 vault_client = SecretClient(vault_url=key_vault_uri, credential=credential)
 retrieved_secret = vault_client.get_secret(secret_name)

I keep getting the error:

azure.core.exceptions.ClientAuthenticationError: Unexpected content type "text/plain; charset=utf-8"
Content: no azure identity found for request clientID

So at some point I attempted to add the managed identity clientID in the cluster secrets and load it from there and still got the same error:

Python code:

    def get_kube_secret(self, secret_name):
        kube_config.load_incluster_config()
        v1_secrets = kube_client.CoreV1Api()
        
        string_secret = str(v1_secrets.read_namespaced_secret(secret_name, "redacted_namespace_name").data).replace("'", "\"")
        json_secret = json.loads(string_secret)
        return json_secret
    
    def decode_base64_string(self, encoded_string):
        decoded_secret = base64.b64decode(encoded_string.strip())
        decoded_secret = decoded_secret.decode('UTF-8')
        return decoded_secret

    managed_identity_client_id_secret = self.get_kube_secret('managed-identity-credential')['clientId']
    managed_identity_client_id = self.decode_base64_string(managed_identity_client_id_secret)

Update:

I also attempted to use the secret store CSI driver, but I have a feeling I'm missing a step there. Should the python code be updated to be able to use the secret store CSI driver?

# This is a SecretProviderClass using user-assigned identity to access the key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-user-msi
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"          # Set to true for using managed identity
    userAssignedIdentityID: "$CLIENT_ID"   # Set the clientID of the user-assigned managed identity to use
    vmmanagedidentityclientid: "$CLIENT_ID"
    keyvaultName: "$KEYVAULT_NAME"        # Set to the name of your key vault
    cloudName: ""                         # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects:  ""
    tenantId: "$AZURE_TENANT_ID"

Deployment Yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: redacted_namespace
  labels:
    app: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: backend
          image: redacted_image
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
          imagePullPolicy: Always
          resources:
            # You must specify requests for CPU to autoscale
            # based on CPU utilization
            requests:
              cpu: "250m"
          env:
            - name:  test-secrets
              valueFrom:
                secretKeyRef:
                  name:  test-secrets
                  key:  test-secrets
          volumeMounts:
            - name: test-secrets
              mountPath: "/mnt/secrets-store"
              readOnly: true
      volumes:
        - name: test-secrets
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "azure-kvname-user-msi"
      dnsPolicy: ClusterFirst

CodePudding user response：

What you are referring to is called pod identity (recently deprecated for workload identity).

if the cluster is configured with managed identity, you can use workload identity.

However, for AKS I suggest configuring the secret store CSI driver to fetch secrets from KV and have them as k8s secrets. To use managed identity for secret provider, refer to this doc.

Then you can configure your pods to read those secrets.

CodePudding user response：

Using the Secrets Store CSI Driver, you can configure the SecretProviderClass to use a workload identity by setting the clientID in the SecretProviderClass. You'll need to use the client ID of your user assigned managed identity and change the usePodIdentity and useVMManagedIdentity setting to false.

With this approach, you don't need to add any additional code in your app to retrieve the secrets. Instead, you can mount a secrets store (using CSI driver) as a volume mount in your pod and have secrets loaded as environment variables which is documented here.

This doc will walk you through setting it up on Azure, but at a high-level here is what you need to do:

Register the EnableWorkloadIdentityPreview feature using Azure CLI
Create an AKS cluster using Azure CLI with the azure-keyvault-secrets-provider add-on enabled and --enable-oidc-issuer and --enable-workload-identiy flags set
Create an Azure Key Vault and set your secrets
Create an Azure User Assigned Managed Identity and set an access policy on the key vault for the the managed identity' client ID
Connect to the AKS cluster and create a Kubernetes ServiceAccount with annotations and labels that enable this for Azure workload identity
Create an Azure identity federated credential for the managed identity using the AKS cluster's OIDC issuer URL and Kubernetes ServiceAccount as the subject
Create a Kubernetes SecretProviderClass using clientID to use workload identity and adding a secretObjects block to enable syncing objects as environment variables using Kubernetes secret store.
Create a Kubernetes Deployment with a label to use workload identity, the serviceAccountName set to the service account you created above, volume using CSI and the secret provider class you created above, volumeMount, and finally environment variables in your container using valueFrom and secretKeyRef syntax to mount from your secret object store.

Hope that helps.