How to define a resource block with for_each that refers to other resources-CodePudding

I create few resources in the terraform -

resource "aws_dms_replication_instance" "foobar_instance_1" {

}

resource "aws_dms_replication_instance" "foobar_instance_2" {

}

resource "aws_dms_endpoint" "foobar_source_1" {

}

resource "aws_dms_endpoint" "foobar_source_2" {

}

Then define a replication task that's dependent on above two resources -

resource "aws_dms_replication_task" "foobar_task_1" {

replication_instance_arn = aws_dms_replication_instance.foobar_instance_1.replication_instance_arn
source_endpoint_arn       = aws_dms_endpoint.foobar_source_1.endpoint_arn
.
.
.
}

I want to use a tf variable to automate a similar aws_dms_replication_task creation using for_each. I create a Map variable in variables.tf that looks like this -

variable "task_map" {
  type        = map(object({
    source_endpoint     = string
    repl_instance     = string
  }))
  default = {
    "dms_attr_map" = {
      source_ep     = "foobar_source_1"
      repl_instance = "foobar_instance_1"
    },
    "dms_attr_map2" = {
      source_ep     = "foobar_source_2"
      repl_instance = "foobar_instance_2"
    }

  }
}

Now, I go on to create a resource block to loop over task_map and create replication_task

resource "aws_dms_replication_task" "for_each_task_1" {
    for_each                  = var.task_map
    replication_instance_arn  =  aws_dms_replication_instance.${each.value["repl_instance"]}.replication_instance_arn
    source_endpoint_arn       = aws_dms_endpoint.${each.value["source_ep"]}.endpoint_arn
    .   
    .
    .
}

As I execute my plan, terrraform plan throws an error that referring to replication_instance using ${each.value["repl_instance"]} is wrong.

Error message -

│ Error: Invalid character │ │ On ../modules/cdc/main.tf line 236: This character is not used within the │ language. ╵

╷ │ Error: Invalid attribute name │ │ On ../modules/cdc/main.tf line 233: An attribute name is required after a │ dot. ╵

Error messages point to the specific line where I use for_each to refer to replication_instance and source_endpoint

aws_dms_endpoint.${each.value["source_ep"]}.endpoint_arn
aws_dms_replication_instance.${each.value["repl_instance"]}.replication_instance_arn

How do I refer to the resources created using their names using a for_each.

Thank you.

CodePudding user response：

How do I refer to the resources created using their names using a for_each.

You can't do this. In other words, you can't create dynamic references to resources in the form of:

aws_dms_endpoint.${each.value["source_ep"]}.endpoint_arn

Instead you have to use maps or lists to create your aws_dms_replication_instance and aws_dms_endpoint. For example:

resource "aws_dms_replication_instance" "foobar_instance" {
  for_each = toset(["foobar_instance_1", "foobar_instance_2"])
}

resource "aws_dms_endpoint" "foobar_source" {
  for_each = toset(["foobar_source_1", "foobar_source_2"])
}

end then refer to them as follows:

resource "aws_dms_replication_task" "for_each_task_1" {
    for_each                  = var.task_map
    replication_instance_arn  = aws_dms_replication_instance.foobar_instance[each.value["repl_instance"]].replication_instance_arn
    source_endpoint_arn       = aws_dms_endpoint.foobar_source[each.value["source_ep"]].endpoint_arn
    .   
    .
    .
}

CodePudding user response：

When thinking about problems like this it's important to recognize that a reference expression like aws_dms_replication_instance.foobar_instance_1 is an indivisible unit: aws_dms_replication_instance alone doesn't exist as a separate data structure that you can query dynamically, because Terraform needs to be able to determine exactly which resources a particular expression depends on in order to produce the dependency graph, before evaluating any expressions.

However, you can construct your own mapping data structure which incorporates the set of resources you are interested in:

locals {
  source_endpoints = {
    "foobar_1" = aws_dms_endpoint.foobar_source_1
    "foobar_2" = aws_dms_endpoint.foobar_source_2
  }
  replication_instances = {
    "foobar_1" = aws_dms_replication_instance.foobar_instance_1
    "foobar_2" = aws_dms_replication_instance.foobar_instance_2
  }
}

You can then use local.source_endpoints and local.replication_instances as mappings to look up the keys specified by your module's caller:

resource "aws_dms_replication_task" "for_each_task_1" {
  for_each = var.task_map

  replication_instance_arn = local.replication_instances[each.value.repl_instance].replication_instance_arn
  source_endpoint_arn      = local.source_endpoints[each.value.source_ep].endpoint_arn

  # ...
}

This can work because a local value is also an object that participates in the dependency graph. Terraform can see that local.source_endpoints depends on both aws_dms_endpoint.foobar_source_1 and aws_dms_endpoint.foobar_source_2, and so therefore indirectly anything which depends on local.source_endpoints must effectively depend on those resources.

Although it's not crucial to your question, I want to note that this design means that your module will declare the full set of aws_dms_endpoint and aws_dms_replication_instance objects, even if some of them don't have any var.task_map elements referring to them.

I wouldn't worry about that if the only practical use of the module involves referring to all of them, but if that isn't true then a variant of this design is to allow the caller to also specify via input variables which endpoints and replication instances they need, and use for_each on all three of these resources.

If you do that then you can avoid constructing the intermediate data structure, because for_each resources are already naturally maps which support the same sort of dynamic lookup:

resource "aws_dms_replication_instance" "example" {
  for_each = var.replication_instances

  # ...
}

resource "aws_dms_endpoint" "example" {
  for_each = var.source_endpoints

  # ...
}

resource "aws_dms_replication_task" "for_each_task_1" {
  for_each = var.task_map

  replication_instance_arn = aws_dms_replication_instance.example[each.value.repl_instance].replication_instance_arn
  source_endpoint_arn      = aws_dms_endpoint.example[each.value.source_ep].endpoint_arn

  # ...
}

In this variant, the caller of the module controls both which replication instances and endpoints exist and which of those each of the tasks use, so the burden for the user of your module is greater but they also get the flexibility of not declaring objects they won't actually use, in case these objects have a significant cost.