I was trying to create a source data set which should point to one of the files in data lake gen2 using terraform. I was using below terraform code.

resource "azurerm_resource_group" "myresourcegroup" {
    name     = "${var.applicationName}-${var.environment}-rg"
    location = var.location
    tags = {
        environment = var.environment
    }
}

Linked service for ADLS gen 2

resource "azurerm_data_factory_linked_custom_service" "storagels" {
  name                 = "${var.applicationName}-${var.environment}-storagels"
  data_factory_id      = azurerm_data_factory.adf.id
  type                 = "AzureBlobFS"
  description          = "storage connection linked service"
  integration_runtime {
    name = azurerm_data_factory_integration_runtime_self_hosted.ir.name
  }
  type_properties_json = <<JSON
  {
        "url": "https://cdfstagesrg.dfs.core.windows.net",
            "accountKey": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "${azurerm_data_factory_linked_service_key_vault.akvls.name}",
                    "type": "LinkedServiceReference"
                },
                "secretName": "storageaccesskey"
            }
        },
       "connectVia":{
      "referenceName": "${azurerm_data_factory_integration_runtime_self_hosted.ir.name}",
      "type": "IntegrationRuntimeReference"
       }    
JSON

  depends_on = [azurerm_data_factory_integration_runtime_self_hosted.ir]
}

Dataset creation

resource "azurerm_data_factory_custom_dataset" "datalakelookup" {
  name            = "${var.dataset_name}-datalakelkp"
  data_factory_id = azurerm_data_factory.adf.id
  type            = "LinkedServiceReference"

  linked_service {
    name = "${var.applicationName}-${var.environment}-storagels"
    
  }
  type_properties_json = <<JSON
  {
    "properties": {
        "linkedServiceName": {
            "referenceName": "${var.applicationName}-${var.environment}-storagels",
            "type": "LinkedServiceReference"
        },
        "annotations": [],
        "type": "Excel",
        "typeProperties": {
            "sheetName": "Datalake",
            "location": {
                "type": "AzureBlobFSLocation",
                "fileName": "tablesnames.xlsx",
                "folderPath": "caseact/Input",
                "fileSystem": "cdfstage"
            },
            "firstRowAsHeader": true
        },
        "schema": []
    }
}
JSON

    depends_on = [azurerm_resource_group.myresourcegroup,azurerm_data_factory.adf,azurerm_data_factory_integration_runtime_self_hosted.ir,azurerm_data_factory_linked_custom_service.clihive-linked-service]
}

All my resources are creating fine but at dataset creation it is giving error as : Error parsing JSON: invalid character 'Â' looking for beginning of value.

Update I have used jsonencode function then getting different error.

CodePudding user response：

Instead of using heredoc syntax for JSON, it would be a better idea to use jsonencode [1]. Here is an example of how you should change the first type_properties_json:

type_properties_json = jsonencode(
  "url" = "https://cdfstagesrg.dfs.core.windows.net"
  "accountKey" = {
     "type" = "AzureKeyVaultSecret"
     "store" = {
        "referenceName" = "${azurerm_data_factory_linked_service_key_vault.akvls.name}",
        "type" = "LinkedServiceReference"
      }
     "secretName" = "storageaccesskey"
    }
   "connectVia" = {
      "referenceName" = "${azurerm_data_factory_integration_runtime_self_hosted.ir.name}"
      "type" = "IntegrationRuntimeReference"
   }
)

Alternatively, I would suggest using the templatefile built-in function [2] in order to make sure you are providing the right values using Terraform interpolation syntax. So, this part from the jsonencode function call would go to a template file (let's call it storagels.tpl):

  "url" = "https://cdfstagesrg.dfs.core.windows.net"
  "accountKey" = {
     "type" = "AzureKeyVaultSecret"
     "store" = {
        "referenceName" = "${account_key_reference_name}",
        "type" = "LinkedServiceReference"
      }
     "secretName" = "storageaccesskey"
    }
   "connectVia" = {
      "referenceName" = "${connect_via_reference_name}"
      "type" = "IntegrationRuntimeReference"
   }

Then, in the type_properties_json, the templatefile function call would be inside of the jsonencode function call:

type_properties_json = jsonencode(templatefile("${path.root}/storagels.tpl",
  account_key_reference_name = azurerm_data_factory_linked_service_key_vault.akvls.name
  connect_via_reference_name = azurerm_data_factory_integration_runtime_self_hosted.ir.name
))

The path.root means that the template file storagels.tpl has to be in the same directory as the terraform code. More about the path variables can be found in [3].

NOTE: You would have to do something similar for the second JSON from your question.

[1] https://www.terraform.io/language/functions/jsonencode

[2] https://www.terraform.io/language/functions/templatefile

[3] https://www.terraform.io/language/expressions/references