Azure bicep Pass Storage Account Connection String to Secret Keyvault loop issue-CodePudding

I have a the following bicep script to perform the following steps:

Create storage accounts based on code
create a key
encrypt storage account with the correct key
generate a secret connection string (storage account) into a key vault

The infra is splitted into 2 resource groups:

rg-shared => where the 2 key vault are (1 keyvault for key and GUID, and 1 keyvault for secrets (connection string)

rg-storage-account => where all the storage account get created

In azure bicep I have the following scripts:

Storage.bicep

param ManagedIdentityid string
param uri string 
param kvname string
param keyvaultrg string = 'XXXX' //<== SHARED Resource Group
var keyVaultKeyPrefix = 'Key-Data-'
var storagePrefix = 'sthritesteur'
param tenantCodes array = [
  'aabb'
  'bbcc'
  'ccdd'
]


// Create storage accounts
resource storageAccount 'Microsoft.Storage/storageAccounts@2021-04-01' = [for tenantCode in tenantCodes: {
  name: '${storagePrefix}${tenantCode}'
  location: resourceGroup().location
  kind: 'StorageV2'
  sku: {
    name: 'Standard_RAGRS'
  }
  // Assign the identity
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
        '${ManagedIdentityid}':{}
    }
  }
  properties: {
    allowCrossTenantReplication: true
    minimumTlsVersion: 'TLS1_2'
    allowBlobPublicAccess: false
    allowSharedKeyAccess: true
    networkAcls: {
      bypass: 'AzureServices'
      virtualNetworkRules: []
      ipRules: []
      defaultAction: 'Allow'
    }
    supportsHttpsTrafficOnly: true
    encryption: {
      identity: {
        // specify which identity to use
        userAssignedIdentity: ManagedIdentityid
      }
      keySource: 'Microsoft.Keyvault'
      keyvaultproperties: {
        keyname: '${kvname}-${keyVaultKeyPrefix}${toUpper(tenantCode)}'
        keyvaulturi:uri
      }
      services: {
        file: {
          keyType: 'Account'
          enabled: true
        }
        blob: {
          keyType: 'Account'
          enabled: true
        }
      }
    }
    accessTier: 'Hot'
  }

}]



resource storage_Accounts_name_default 'Microsoft.Storage/storageAccounts/blobServices@2021-04-01' = [ for (storageName, i) in tenantCodes :{
  parent: storageAccount[i]
  name: 'default'
  properties: {
    changeFeed: {
      enabled: false
    }
    restorePolicy: {
      enabled: false
    }
    containerDeleteRetentionPolicy: {
      enabled: true
      days: 7
    }
    cors: {
      corsRules: []
    }
    deleteRetentionPolicy: {
      enabled: true
      days: 30
    }
    isVersioningEnabled: true
  }
}]

module connectionString 'shared.bicep' = [for (storageName, i) in tenantCodes :{
  scope: resourceGroup(keyvaultrg)
  name: storageName
  params: {
    storageAccountString: 'DefaultEndpointsProtocol=https;AccountName=${storageAccount[i].name};AccountKey=${listKeys(storageAccount[i].id, storageAccount[i].apiVersion).keys[0].value};EndpointSuffix=${environment().suffixes.storage}'
  }
}]

keyvaultclient.bicep

param deploymentIdOne string = newGuid()
param deploymentIdTwo string = newGuid()
output deploymentIdOne string = '${deploymentIdOne}-${deploymentIdTwo}'
output deploymentIdTwo string = deploymentIdTwo

param storagerg string = 'XXXX' //<=== Storage Accounts Resource Groups
param sharedManagedIdentity string = 'mgn-identity-shared'
param keyvaultmain string = 'XXXX' //<=== KeyVault Name where to create GUID AND Keys
param tenantCodes array = [
  'aabb'
  'bbcc'
  'ccdd'
]
var clientDataKeyPrefix = 'Key-Data-'

resource managedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2018-11-30' = {
  name: sharedManagedIdentity
  location: resourceGroup().location
}

resource keyVaultClients 'Microsoft.KeyVault/vaults@2021-06-01-preview' existing = {
  name: keyvaultmain
}
resource kvClientsKey 'Microsoft.KeyVault/vaults/keys@2021-06-01-preview' = [for code in tenantCodes: {
  parent:keyVaultClients
  name: '${keyVaultClients.name}-${clientDataKeyPrefix}${toUpper(code)}'
  properties: {
    keySize: 2048
    kty: 'RSA'
    // Assign the least permission
    keyOps: [
      'unwrapKey'
      'wrapKey'
    ]
  }
}]
resource accessPolicy 'Microsoft.KeyVault/vaults/accessPolicies@2021-06-01-preview' = {
  parent:keyVaultClients
  name: 'add'
  properties: {
    accessPolicies: [
      {
        tenantId: subscription().tenantId
        objectId: managedIdentity.properties.principalId
        permissions: {
          // minimum required permission
          keys: [
            'get'
            'unwrapKey'
            'wrapKey'
          ]
        } 
      }
    ]
  }
}
resource clientLearnersGuid 'Microsoft.KeyVault/vaults/secrets@2021-06-01-preview' = [for tenant in tenantCodes: {
  parent:keyVaultClients
  name: '${keyVaultClients.name}${tenant}'
  properties: {
    contentType: 'GUID Key'
    value: '${deploymentIdOne}-${deploymentIdTwo}'
  }
  dependsOn:kvClientsKey
}]
module StorageAccount 'storage.bicep' = [for (storageName, i) in tenantCodes :{
  scope: resourceGroup(storagerg)
  name: storageName
  params: {
    ManagedIdentityid:managedIdentity.id
    kvname:keyVaultClients.name
    uri:keyVaultClients.properties.vaultUri
  }
  dependsOn:clientLearnersGuid
}]

shared.bicep

param keyvaultshared string = 'XXXX' //<=== Key Vault Where to Store the Storage Connection String Secret
param storageAccountString string
param tenantCodes array = [
  'aabb'
  'bbcc'
  'ccdd'
]
resource keyVaultShared 'Microsoft.KeyVault/vaults@2021-06-01-preview' existing = {
  name: keyvaultshared
}
resource storageAccountConnectionString 'Microsoft.KeyVault/vaults/secrets@2021-06-01-preview' = [for tenant in tenantCodes: {
  parent:keyVaultShared
  name: '${keyVaultShared.name}-test${tenant}'
  properties:{
    contentType: '${tenant} Storage Account Connection String'
    value: storageAccountString
  }
}]

Those script performs all the steps I needed based on the tenantCode. Which is just perfect. If I have only 1 tenantCode declared, everything goes smooth and perfect, but the issue I am facing raises when I try to declare more than 1. And this is the problem in details.

When I declare more than 1 code, the script still created all the resources I needed: Storage accounts,Keys,Encryptions, GUID and ConnectionStrings Secrets. But it fails anyway on the ConnectionStrings Secret.

The reason why it fails it because in those files at this block of code:

Shared.bicep

resource storageAccountConnectionString 'Microsoft.KeyVault/vaults/secrets@2021-06-01-preview' = [for tenant in tenantCodes: {
  parent:keyVaultShared
  name: '${keyVaultShared.name}-test${tenant}'
  properties:{
    contentType: '${tenant} Storage Account Connection String'
    value: storageAccountString
  }
}]

and

keuvaultsclient.bicep*

module StorageAccount 'storage.bicep' = [for (storageName, i) in tenantCodes :{
  scope: resourceGroup(storagerg)
  name: storageName
  params: {
    ManagedIdentityid:managedIdentity.id
    kvname:keyVaultClients.name
    uri:keyVaultClients.properties.vaultUri
  }
  dependsOn:clientLearnersGuid
}]

I have a multiple loop, on which I realised that in my shared keyvault under secrets, I have the correct amount of secrets (with 3 tenant codes I have 3 secrets) and under each secret I have 3 versions (for each tenant code it generate a new version for each secret). This looping error causes bicep script to fail with the following message:

{"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"Conflict","message":"{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"Conflict\",\r\n            \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"StorageAccountOperationInProgress\\\",\\r\\n    \\\"message\\\": \\\"An operation is currently performing on this storage account that requires exclusive access.\\\"\\r\\n  }\\r\\n}\"\r\n          },\r\n          {\r\n            \"code\": \"Conflict\",\r\n            \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"StorageAccountOperationInProgress\\\",\\r\\n    \\\"message\\\": \\\"An operation is currently performing on this storage account that requires exclusive access.\\\"\\r\\n  }\\r\\n}\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"}]}}

I am totally blocked at this stage and as I am a bicep beginner, I run out of all options and test to try to solve this issue.

How to reproduce

Create 2 resource groups(one for the key vaults and one for the storage accounts)
Update parameters with the correct kv names
In all files add atleast 2 tenantCode on each file
execute the command az deployment group create -f ./keyvaultsclient.bicep -g <rg-where-keyvaults-are>

I hope I explained clearly enough the issue I am facing and please if you need more details just let me know.

Thank you so much for your time and help

CodePudding user response：

I tested the same code by doing some changes, Please try doing the same changes in the .bicep files :

keyvaultclient.bicep:

Removed the loop for module as its creating 2 modules for the same thing.

module StorageAccount './storage.bicep' = {
  scope: resourceGroup(storagerg)
  name: 'NestedStorage'
  params: {
    ManagedIdentityid:managedIdentity.id
    kvname:keyVaultClients.name
    uri:keyVaultClients.properties.vaultUri
  }
  dependsOn:clientLearnersGuid
}

storage.bicep:

Removed the loop for module and added the loop for only storage connection string which will store the outputs in array and pass it to the next module.

module connectionString './shared.bicep'={
  scope: resourceGroup(keyvaultrg)
  name: 'KeyvaultNested'
  params: {
    storageAccountString: [for (tenant,i) in tenantCodes :{
    id:'DefaultEndpointsProtocol=https;AccountName=${storageAccount[i].name};AccountKey=${listKeys(storageAccount[i].id, storageAccount[i].apiVersion).keys[0].value};EndpointSuffix=${environment().suffixes.storage}'
  }]
}
}

shared.bicep:

Changed the parameter type of StorageAccountString from string to array and added [for (tenant,i) in tenantCodes in the secret part so that I can give the value as storageAccountString[i].id .

param storageAccountString array

resource storageAccountConnectionString 'Microsoft.KeyVault/vaults/secrets@2021-06-01-preview' = [for (tenant,i) in tenantCodes: {
  parent:keyVaultShared
  name: '${keyVaultShared.name}-test${tenant}'
  properties:{
    contentType: '${tenant} Storage Account Connection String'
    value: storageAccountString[i].id
  }
}]

Outputs: