Description

I'm attempting to iterate on the results of a terraform module and build a set of AWS glue crawlers and IAM roles based on how many source S3 buckets are set in the terraform module

I'm specifically trying to use this line

  for_each      = {for key, value in setproduct(local.bucket-names, module.crawler_source_buckets.id): key=>value}

or this line

  for_each      = {for key, value in setproduct(local.bucket-names, module.crawler_source_buckets.*.id): key=>value}

or this line

  for_each      = {for key, value in setproduct(local.bucket-names, module.crawler_source_buckets.[*].id): key=>value}

but these all result in different errors such as

│ Error: Invalid function argument
│
│   on catalog.tf line 11, in resource "aws_glue_crawler" "bucket_crawlers":
│   11:   for_each      = {for key, value in setproduct(local.bucket-names, module.crawler_source_buckets.id): key=>value}
│     ├────────────────
│     │ while calling setproduct(sets...)
│     │ module.crawler_source_buckets.id is a object
│
│ Invalid value for "sets" parameter: a set or a list is required.

or

│ Error: Invalid template interpolation value
│
│   on catalog.tf line 14, in resource "aws_glue_crawler" "bucket_crawlers":
│   14:     path = "s3://${each.value[1]}"
│     ├────────────────
│     │ each.value[1] is object with 2 attributes
│
│ Cannot include the given value in a string template: string required.

which is making me question if the thing I want to do is even possible. Maybe the answer in this case is that I use the base variable declarations since AWS S3 Ids and ARNs follow a known standard pattern but I would like to know how to iterate on module outputs that are declared with a for_each pattern since that seems very useful.

Code

s3.tf

locals {
  bucket-names = [
    "${var.env_name}-failed-stripe-payment-reports",
    "${var.env_name}-fraud-log-return-responses"]
}


module "crawler_source_buckets" {
  source      = "../../modules/s3-bucket-v2"
  for_each     = toset(local.bucket-names)

  bucket_name = each.value
  # There's more code but I'm choosing to leave it out for clairty
]
}

s3-bucket-v2 output.tf

output "arn" {
  description = "The ARN of the main bucket."
  value       = aws_s3_bucket.main.arn
}


output "id" {
  description = "The ID (name) of the main bucket."
  value       = aws_s3_bucket.main.id
}
# There's an extra 60 lines I'm choosing to leave out for clarity.

catalog.tf

resource "aws_glue_catalog_database" "catalog_databases" {
  name = each.value
  for_each     = toset(local.bucket-names)
}


resource "aws_glue_crawler" "bucket_crawlers" {
  database_name = each.value[0]
  name          = "${each.value[0]}-crawler"
  role          = aws_iam_role.glue_crawler_role.arn
  for_each      = {for key, value in setproduct(local.bucket-names, module.crawler_source_buckets.id): key=>value}

  s3_target {
    path = "s3://${each.value[1]}"
  }
  schedule = "cron(*/15 * * * ? *)"
}

CodePudding user response：

Since you are using for_each in the module you have to access it using key. For example, module.crawler_source_buckets["key"].id. To get all ids you have to use the following:

values(module.crawler_source_buckets)[*].id

So it should be:

for_each      = {for key, value in setproduct(local.bucket-names, values(module.crawler_source_buckets)[*].id): key=>value}