How do I join multiple JSON files for table mapping on AWS DMS using Terraform?-CodePudding

In order to create a replication task of DMS, I use this resource:

resource "aws_dms_replication_task" "test" {
  migration_type            = "full-load"
  replication_instance_arn  = aws_dms_replication_instance.test-dms-replication-instance-tf.replication_instance_arn
  replication_task_id       = "test-dms-replication-task-tf"
  source_endpoint_arn       = aws_dms_endpoint.test-dms-source-endpoint-tf.endpoint_arn
  table_mappings           = file("${path.module}/db1/test1.json")

  tags = {
    Name = "test"
  }

  target_endpoint_arn = aws_dms_endpoint.test-dms-target-endpoint-tf.endpoint_arn
}

There are many JSON files generated by the schema, such as:

./db1/test1.json

{
  "rules": [
    {
      "rule-type": "transformation",
      "rule-id": "1",
      "rule-name": "Rule name 1",
      "rule-action": "rename",
      "rule-target": "schema",
      "object-locator": {
        "schema-name": "SCHEMA_1",
        "table-name": "TABLE_1"
      },
      "value": "main"
    }
  ]
}

./db1/test2.json

{
  "rules": [
    {
      "rule-type": "transformation",
      "rule-id": "2",
      "rule-name": "Rule name 2",
      "rule-action": "rename",
      "rule-target": "schema",
      "object-locator": {
        "schema-name": "SCHEMA_2",
        "table-name": "TABLE_2"
      },
      "value": "main"
    }
  ]
}

Finally, I want to get a full rules JSON output like

{
  "rules": [
    {
      "rule-type": "transformation",
      "rule-id": "1",
      "rule-name": "Rule name 1",
      "rule-action": "rename",
      "rule-target": "schema",
      "object-locator": {
        "schema-name": "SCHEMA_1",
        "table-name": "TABLE_1"
      },
      "value": "main"
    },
    {
      "rule-type": "transformation",
      "rule-id": "2",
      "rule-name": "Rule name 2",
      "rule-action": "rename",
      "rule-target": "schema",
      "object-locator": {
        "schema-name": "SCHEMA_2",
        "table-name": "TABLE_2"
      },
      "value": "main"
    }
  ]
}

It seems that I need to merge the child elements.

How do I join these files together without putting them manually inside 1 JSON file using Terraform?

CodePudding user response：

There are probably several ways of doing this. One would be as follows:

locals {

    rule_files = ["${path.module}/test1.json", "${path.module}/test2.json"]

    rules_joined = {
                    rules = flatten([for fname in local.rule_files: jsondecode(file(fname))["rules"]])
                   }
}

which gives local.rules_joined as:

{
  "rules" = [
    {
      "object-locator" = {
        "schema-name" = "SCHEMA_1"
        "table-name" = "TABLE_1"
      }
      "rule-action" = "rename"
      "rule-id" = "1"
      "rule-name" = "Rule name 1"
      "rule-target" = "schema"
      "rule-type" = "transformation"
      "value" = "main"
    },
    {
      "object-locator" = {
        "schema-name" = "SCHEMA_2"
        "table-name" = "TABLE_2"
      }
      "rule-action" = "rename"
      "rule-id" = "2"
      "rule-name" = "Rule name 2"
      "rule-target" = "schema"
      "rule-type" = "transformation"
      "value" = "main"
    },
  ]
}

To make it into a string, use jsonencode(local.rules_joined).