Terraform - /usr/local/bin/jq: Argument list too long-CodePudding

This is a variation on a question that's been asked before.

I'm using an external data source in Terraform to ask it for a list of volume snapshots in AWS Dublin, and JQ in a templatefile to extract the snapshot ids.

data "external" "volsnapshot_ids" {

  program    = [

    "bash",

    "-c",

    templatefile("cli.tftpl", {input_string = "aws ec2 describe-snapshots --region=eu-west-1", top = "Snapshots", next = "| .SnapshotId"})]

}

And it uses this templatefile:

#!/bin/bash

set -e

OUTPUT=$(${input_string} | jq  -r -c '.${top}[] ${next}' | jq -R -s -c 'split("\n")' | jq '.[:-1]')

jq -n -c --arg output "$OUTPUT" '{"output":$output}'

The basic CLI command with JQ works and looks like this:

aws ec2 describe-snapshots --region=eu-west-1 | jq  -r -c '.Snapshots[] | .SnapshotId' | jq -R -s -c 'split("\n")' | jq '.[:-1]' | wc -l

It returns a lot of snapshot ids.

When I run it through Terraform though, it errors:

Error: External Program Execution Failed

│ 

│   with data.external.volsnapshot_ids,

│   on data.tf line 304, in data "external" "volsnapshot_ids":

│  304:   program    = [

│  305:     "bash",

│  306:     "-c", 

│  307:     templatefile("cli.tftpl", {input_string = "aws ec2 describe-snapshots --region=eu-west-1", top = "Snapshots", next = "| .SnapshotId"})]

│ 

│ The data source received an unexpected error while attempting to execute

│ the program.

│ 

│ Program: /bin/bash

│ Error Message: bash: line 6: /usr/local/bin/jq: Argument list too long

│ 

│ State: exit status 1

I think it's the size of the dataset being returned because it works in regions with less snapshot ids - London works.

Sizewise, here's London:

aws ec2 describe-snapshots --region=eu-west-2 | jq  -r -c '.Snapshots[] | .SnapshotId' | jq -R -s -c 'split("\n")' | jq '.[:-1]' | wc -l
20000

And here's Dublin:

aws ec2 describe-snapshots --region=eu-west-1 | jq  -r -c '.Snapshots[] | .SnapshotId' | jq -R -s -c 'split("\n")' | jq '.[:-1]' | wc -l
42500

Is there a way to fix up the JQ in my templatefile so it can handle big JSON files?

CodePudding user response：

I wouldn't recommend using command inside TF datasource. Might be hard to debug. There is a data_source for EBS snapshots.

As for your command inside template, in order to debug it you need to simulate the same environment. E.g. instead of running as is, try to repeat what you have in template, like bash -c and so on. Also you can add output to see the template rendered to see if there are any issues.

CodePudding user response：

Scroll to bottom of answer.

Don't provide the value as argument, but via directly standard input:
aws ... \ | jq -rc '.${top}[] ${next}' \ | jq -Rsc './"\n"' \ | jq -c '.[:-1]' | jq -Rc '{output:.}'

~~Note that you are can probably combine most of the separate jq invocations into a single jq program.~~

This pipeline of jq invocations is a massively, massively overcomplicated non-solution. Why convert back and forth between strings and JSON objects, parsing those strings again, when jq can already process the data directly?

aws ... | jq -c '{ output: .Snapshots | map(.SnapshotId) | tostring }'

Example output:

{"output":"[\"snap-cafebabe\",\"snap-deadbeef\",\"snap-0123abcd\"]"}

If you have to use variables:

top=Snapshots
next=SnapshotId
aws ... | jq --arg top "$top" --arg next "$next" -c '{ output: .[$top] | map(.[$next]) | tostring }'