How to securely allow access to AWS Secrets Manager with Terraform and cloud-init-CodePudding

I have the situation whereby I am having Terraform create a random password and store it into AWS Secrets Manager. My Terraform password and secrets manager config:

resource "random_password" "my_password" {
  length = 16
  lower = true
  upper = true
  number = true
  special = true
  override_special = "@#$%"
}

resource "aws_secretsmanager_secret" "my_password_secret" {
  name = "/development/my_password"
}

resource "aws_secretsmanager_secret_version" "my_password_secret_version" {
  secret_id     = aws_secretsmanager_secret.my_password_secret.id
  secret_string = random_password.my_password.result
}

The above works well. However I am not clear on how to achieve my final goal...

I have an AWS EC2 Instance which is also configured via Terraform, when the system boots it executes some cloud-init config which runs a setup script (Bash script). The Bash setup script needs to install some server software and set a password for that server software. I am not certain how to securely access my_password from that Bash script during setup.

My Terraform config for the instance and cloud-init config:

resource "aws_instance" "my_instance_1" {
  ami           = data.aws_ami.amazon_linux_2.id
  instance_type = "m5a.2xlarge"

  user_data = data.cloudinit_config.my_instance_1.rendered

  ...

}


data "cloudinit_config" "my_instance_1" {
  gzip = true
  base64_encode = true

  part {
    content_type = "text/x-shellscript"
    filename = "setup-script.sh"
    content = <<EOF
#!/usr/bin/env bash
my_password=`<MY PASSWORD IS NEEDED HERE>`  # TODO retrieve via cURL call to Secrets Manager API?
server_password=$my_password /opt/srv/bin/install.sh
EOF
  }
}

I need to be able to securely retrieve the password from the AWS Secrets Manager when the cloud-init script runs, as I have read that embedding it in the bash script is considered insecure.

I have also read that AWS has the notion of Temporary Credentials, and that these can be associated with an EC2 instance - https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html

Using Terraform can I create temporary credentials (say 10 minutes TTL) and grant them to my AWS EC2 instance, so that when my Bash script runs during cloud-init it can retrieve the password from the AWS Secrets Manager?

I have seen that on the Terraform aws_instance resource, I can associate a iam_instance_profile and I have started by trying something like:

resource "aws_iam_instance_profile" "my_instance_iam_instance_profile" {
  name = "my_instance_iam_instance_profile"
  path = "/development/"
  
  role = aws_iam_role.my_instance_iam_role.name
  
  tags = {
    Environment = "dev"
  }
}

resource "aws_iam_role" "my_instance_iam_role" {
  name = "my_instance_iam_role"
  path = "/development/"

  // TODO - what how to specify a temporary credential access to a specific secret in AWS Secrets Manager from EC2???

  tags = {
    Environment = "dev"
  }
}

resource "aws_instance" "my_instance_1" {
  ami           = data.aws_ami.amazon_linux_2.id
  instance_type = "m5a.2xlarge"

  user_data = data.cloudinit_config.my_instance_1.rendered

  iam_instance_profile = join("", [aws_iam_instance_profile.my_instance_iam_instance_profile.path, aws_iam_instance_profile.my_instance_iam_instance_profile.name])

  ...

}

Unfortunately I can't seem to find any details on what I should put in the Terraform aws_iam_role which would allow my EC2 instance to access the Secret in the AWS Secrets Manager for a temporary period of time.

Can anyone advise? I would also be open to alternative approaches as long as they are also secure.

Thanks

CodePudding user response：

You can create aws_iam_policy or an inline policy which can allow access to certain SSM parameters based on date and time.

In case of inline policy, this can be attached to the instance role which would look something like this:

resource "aws_iam_role" "my_instance_iam_role" {
  name = "my_instance_iam_role"
  path = "/development/"

  inline_policy {
    name = "my_inline_policy"

    policy = jsonencode({
       "Version": "2012-10-17",
       "Statement": [{
           "Effect": "Allow",
           "Action": "ssm:GetParameters",
           "Resource": "arn:aws:ssm:us-east-2:123456789012:parameter/development-*",
           "Condition": {
               "DateGreaterThan": {"aws:CurrentTime": "2020-04-01T00:00:00Z"},
               "DateLessThan": {"aws:CurrentTime": "2020-06-30T23:59:59Z"}
           }
       }]
    })
  }
  tags = {
    Environment = "dev"
  }
}

CodePudding user response：

There are two main ways to achieve this:

pass the value as is during the instance creation with terraform
post-bootstrap invocation of some script

Your approach of polling it in the cloud-init is a hybrid one, which is perfectly fine, but I'm not sure whether you actually need to go down that route.

Let's explore the first option, where you do everything in terraform. We have two sub-options there depending on where you create the secret and the instance within the same terraform execution run (within the same folder in which the code resides) or it's a two step process, where you create the secret first, and then the instance, as it has a minor difference between the two on how to pass the secret value as a var to the script.

Case A: in case they are created together:

You can pass the password directly to the script.

resource "random_password" "my_password" {
  length = 16
  lower = true
  upper = true
  number = true
  special = true
  override_special = "@#$%"
}

resource "aws_secretsmanager_secret" "my_password_secret" {
  name = "/development/my_password"
}

resource "aws_secretsmanager_secret_version" "my_password_secret_version" {
  secret_id     = aws_secretsmanager_secret.my_password_secret.id
  secret_string = random_password.my_password.result
}

data "cloudinit_config" "my_instance_1" {
  gzip = true
  base64_encode = true

  part {
    content_type = "text/x-shellscript"
    filename = "setup-script.sh"
    content = <<EOF
#!/usr/bin/env bash
server_password=${random_password.my_password.result} /opt/srv/bin/install.sh
EOF
  }
}

Case B: if they are created in separate folders

You could use a data resource to get the secret value from terraform (the role with which you are deploying your terraform code will need permissions GetSecret)

data "aws_secretsmanager_secret_version" "my_password" {
   secret_id = "/development/my_password"
}

data "cloudinit_config" "my_instance_1" {
  gzip = true
  base64_encode = true

  part {
    content_type = "text/x-shellscript"
    filename = "setup-script.sh"
    content = <<EOF
#!/usr/bin/env bash
server_password=${data.aws_secretsmanager_secret_version.my_password.secret_string} /opt/srv/bin/install.sh
EOF
  }
}

In both cases you wouldn't need to assign SSM permissions to the EC2 instance profile attached to the instance, you won't need to use curl or other means in the script, and the password would not be part of your bash script.

It will be stored in your terraform state, so you should make sure that the access to it is restricted. Even with the hybrid approach where you are going to get the secret from the secret manager during the instance bootstrap, the password would still be stored in your state as you are creating that secret with resource "random_password" as per the Terraform Sensitive data in state.

Now, let's look at option 2. It is very similar to your approach, but instead of doing it in the user-data, you can use Systems Manager Run Command to start your installation script as a post-bootstrap step. Then depending on how do you invoke the script, whether it is present locally on the instance, or you are using a document with a State Manager you can either pass the secret to it as a variable again, or get it from the Secrets Manager with aws-cli or curl, or whatever you prefer (which will require the necessary level of IAM permissions).