Generate terraform variables file from shell/bash script-CodePudding

my requirement is to generate the terraform variables file based on the environment, I'm trying to generate them using bash/shell script, but having difficulty converting the output to terraform HCL language (I can't use JSON because I'm manipulating them further in terraform module)

Current API output (which needs to be converted into HCL):

'cluster_name1' 'REGION1' 'Volume_Size1' 'Instance_Size1'
'cluster_name2' 'REGION2' 'Volume_Size2' 'Instance_Size2'
'cluster_name3' 'REGION3' 'Volume_Size3' 'Instance_Size3'
{...}

Output in CSV format:

"cluster_name1","REGION1","Volume_Size1","Instance_Size1"
"cluster_name2","REGION2","Volume_Size2","Instance_Size2"
"cluster_name3","REGION3","Volume_Size3","Instance_Size3"
{...}

Required format:

variable "cluster_configuration" {
    default = [
    {   
        "cluster_name" : "cluster_name1"
        "cluster_region" : "REGION1"
        "instance_db_size" : "Volume_Size1"
        "instance_size" : "Instance_Size1"
    },
    {
        "cluster_name" : "cluster_name2"
        "cluster_region" : "REGION2"
        "instance_db_size" : "Volume_Size2"
        "instance_size" : "Instance_Size2"
    },
    {
        "cluster_name" : "cluster_name3"
        "cluster_region" : "REGION3"
        "instance_db_size" : "Volume_Size3"
        "instance_size" : "Instance_Size3"
    },
    {....}
    ]
}

My Terraform code, just for reference:

locals {
  dbconfig = [
    for db in var.cluster_configuration : [{
        instance_name = db.cluster_name
        db_size = db.instance_db_size
        instance_size = db.instance_size
        cluster_region = db.cluster_region
      }
    ]
  ]
}

I have tried with AWK and SED but have had no luck so far.

CodePudding user response：

Terraform can process JSON data, if you define the objects correctly. You might be able to use jq to format the given CSV data into appropriate JSON for Terraform to consume. If I recall correctly, the filter would be something like

# This *very* much assumes that the quotes aren't actually
# quoting fields that contain a real comma. jq is not suitable
# for robustly parsing CSV data.
[
  inputs |
     split(",") |
     map(ltrimstr("\"")) |
     map(rtrimstr("\"")) |
     {
        cluster_name: .[0],
        cluster_region: .[1],
        instance_db_size: .[2],
        instance_size: .[3]
     }
] | {variable: {cluster_configuration: {default: .}}}

(There's probably room for improvement.) Assuming you save that to a file like api.jq and your API output is in output.csv, then

$ jq -nRf api.jq output.csv
{
  "variable": {
    "cluster_configuration": {
      "default": [
        {
          "cluster_name": "cluster_name1",
          "cluster_region": "REGION1",
          "instance_db_size": "Volume_Size1",
          "instance_size": "Instance_Size1"
        },
        {
          "cluster_name": "cluster_name2",
          "cluster_region": "REGION2",
          "instance_db_size": "Volume_Size2",
          "instance_size": "Instance_Size2"
        },
        {
          "cluster_name": "cluster_name3",
          "cluster_region": "REGION3",
          "instance_db_size": "Volume_Size3",
          "instance_size": "Instance_Size3"
        }
      ]
    }
  }
}

It might be simpler to pick the language of your choice with a proper CSV parser, though, to generate the JSON.

CodePudding user response：

Solution using jq.

The content is as requested (but the specified formatting is disregarded)

INPUT="
'cluster_name1' 'REGION1' 'Volume_Size1' 'Instance_Size1'
'cluster_name2' 'REGION2' 'Volume_Size2' 'Instance_Size2'
'cluster_name3' 'REGION3' 'Volume_Size3' 'Instance_Size3'
"

jq -srR '
  split("\n") |        # split lines
  map(split(" ") |     # split fields
      select(any) |    # remove emty lines
      map(.[1:-1]) |   # remove enclosing quotes
      {
        cluster_name: .[0],
        cluster_region: .[1],
        instance_db_size: .[2],
        instance_size: .[3]
      }) |
  "variable \"cluster_configuration\" {",
  "    default = ",
  .,
  "}"
' <<< "$INPUT"

Output

variable "cluster_configuration" {
    default = 
[
  {
    "cluster_name": "cluster_name1",
    "cluster_region": "REGION1",
    "instance_db_size": "Volume_Size1",
    "instance_size": "Instance_Size1"
  },
  {
    "cluster_name": "cluster_name2",
    "cluster_region": "REGION2",
    "instance_db_size": "Volume_Size2",
    "instance_size": "Instance_Size2"
  },
  {
    "cluster_name": "cluster_name3",
    "cluster_region": "REGION3",
    "instance_db_size": "Volume_Size3",
    "instance_size": "Instance_Size3"
  }
]
}