Trying to create an AWS Batch Job Definition via Terraform.
Json used in the data source template_file:
[
{
"command": "${docker_command}",
"image": "${docker_image}",
"resourceRequirements": [
{
"type": "MEMORY",
"value": "${memory}"
},
{
"type": "VCPU",
"value": "${vcpu}"
}
],
"jobRoleArn": "${job_role_arn}",
"volumes": [
{
"host": {
"sourcePath": "/mnt/efs"
},
"name": "efs"
}
],
"mountPoints": [
{
"sourceVolume": "efs",
"containerPath": "/mnt/efs",
"readOnly": false
}
]
}
]
Terraform code:
data "template_file" "dev_jd_template" {
template = "${file("./data/job_definition.json")}"
vars = {
docker_command = "[python /app/data_migrator.py]"
docker_image = "python:3.9.9-slim-bullseye"
memory = 1024
vcpu = 1
job_role_arn = "arn:aws:iam::xxxxxxxx:role/AWSBatchJobRole"
}
}
resource "aws_batch_job_definition" "job_definition" {
name = "dev-jd-3"
type = "container"
platform_capabilities = ["EC2"]
container_properties = "${data.template_file.dev_jd_template.rendered}"
}
Error:
AWS Batch Job container_properties is invalid: Error decoding JSON: json: cannot unmarshal array into Go value of type batch.ContainerProperties
│
│ with aws_batch_job_definition.job_definition,
│ on jd.tf line 23, in resource "aws_batch_job_definition" "job_definition":
│ 23: container_properties = "${data.template_file.dev_jd_template.rendered}"
If I pass the value for container_properties as file("./data/job_definition.json"), it works fine but as we have more than 1 job_definitions, we were hoping to use template.
I have also looked at a similar SO question here but couldn't figure out what's wrong in my case.
TIA!
CodePudding user response：
It took a while to realize where the issue is because it is not that obvious and the error is not of too much help. Now, looking at the example from the official documentation [1], here is where the catch is:
"command": ["ls", "-la"],
This is expected for the command argument. So it is actually a list of strings. In your question, you are using a string which looks like a list consisting of only one element:
"[python /app/data_migrator.py]"
The above would be a single line, a single string, not a list of strings as required by the command property. Additionally, this seems to be an edge case since you cannot pass a list of strings neither using template_file data source nor templatefile built-in function the way you have tried to.
With some amount of poking around and using an example from the Terraform docs [2], I have managed to get an output that you want. First, the change that needs to happen in the templated file:
${jsonencode({
"command": [ for command in docker_command: command ],
"image": docker_image,
"resourceRequirements": [
{
"type": "MEMORY",
"value": memory
},
{
"type": "VCPU",
"value": vcpu
}
],
"jobRoleArn": job_role_arn,
"volumes": [
{
"host": {
"sourcePath": "/mnt/efs"
},
"name": "efs"
}
],
"mountPoints": [
{
"sourceVolume": "efs",
"containerPath": "/mnt/efs",
"readOnly": false
}
]
})}
This will replace all the placeholder values in the template with the values provided when calling the templatefile function. Note that you actually have to use double quotes around values for CPU and memory because otherwise you get the following error:
AWS Batch Job container_properties is invalid: Error decoding JSON: json: cannot unmarshal number into Go struct field ResourceRequirement.ResourceRequirements.Value of type string
This is fine because the AWS docs say that those values are strings anyway [3].
Now that the template is fixed, here is how to use the templatefile built-in [4] function:
resource "aws_batch_job_definition" "job_definition" {
name = "dev-jd-3"
type = "container"
platform_capabilities = ["EC2"]
container_properties = templatefile("${path.root}/data/job_definition.json",
{
docker_command = ["python", "/app/data_migrator.py"]
docker_image = "python:3.9.9-slim-bullseye"
memory = "1024" # <----- quoted value!
vcpu = "1" # <----- quoted value!
job_role_arn = "arn:aws:iam::xxxxxxxx:role/AWSBatchJobRole"
})
}
This should result in a plan like this:
Terraform will perform the following actions:
# aws_batch_job_definition.job_definition will be created
resource "aws_batch_job_definition" "job_definition" {
arn = (known after apply)
container_properties = jsonencode(
{
command = [
"python",
"/app/data_migrator.py",
]
image = "python:3.9.9-slim-bullseye"
jobRoleArn = "arn:aws:iam::xxxxxxxx:role/AWSBatchJobRole"
mountPoints = [
{
containerPath = "/mnt/efs"
readOnly = false
sourceVolume = "efs"
},
]
resourceRequirements = [
{
type = "MEMORY"
value = "1024"
},
{
type = "VCPU"
value = "1"
},
]
volumes = [
{
host = {
sourcePath = "/mnt/efs"
}
name = "efs"
},
]
}
)
id = (known after apply)
name = "dev-jd-3"
platform_capabilities = [
"EC2",
]
propagate_tags = false
revision = (known after apply)
tags_all = (known after apply)
type = "container"
}
I would also strongly recommend renaming the template file to use something like job_definition.json.tftpl naming convention, as this is suggested in the documentation:
*.tftplis the recommended naming pattern to use for your template files. Terraform will not prevent you from using other names, but following this convention will help your editor understand the content and likely provide better editing experience as a result.
[3] https://docs.aws.amazon.com/batch/latest/APIReference/API_ResourceRequirement.html
[4] https://developer.hashicorp.com/terraform/language/functions/templatefile