Remove last `},` in a dynamically generated JSON array using sed or jq-CodePudding

I am attempting to dynamically generate a JSON, however, the method I follow leaves a last }, making the JSON invalid:

#Initializing Current State file
    echo "[" > state.json
    # Getting list of all EKS clusters
    for cluster in $(aws eks list-clusters --output text | awk '{print $2}')
    do
      echo "====== $cluster ======"
      # Get list of all ASG from cluster
      for asg in $(aws autoscaling describe-auto-scaling-groups  --query 'AutoScalingGroups[?contains(Tags[?Key==`eks:cluster-name`].Value, `'"$cluster"'`)].[AutoScalingGroupName]' --output text)
      do
        echo "Getting capacity for ASG: $asg"
        MinSize=$(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name $asg --query 'AutoScalingGroups[*].MinSize' --output text)
        DesiredCapacity=$(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name $asg --query 'AutoScalingGroups[*].DesiredCapacity' --output text)  
        MaxSize=$(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name $asg --query 'AutoScalingGroups[*].MaxSize' --output text)
        JSON_STRING=$( jq -n \
                  --arg cn "$cluster" \
                  --arg asg "$asg" \
                  --arg MinSize "$MinSize" \
                  --arg DesiredCapacity "$DesiredCapacity" \
                  --arg MaxSize "$MaxSize" \
                  '{cluster: $cn, asgname: $asg, minSize: $MinSize, DesiredCapacity: $DesiredCapacity, MaxSize: $MaxSize}' )
        echo "${JSON_STRING}," >> state.json
      done
    done
    #Finalizing Current State file
    echo "]" >> state.json

$ cat state.json
[
{
  "cluster": "eks-b",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
},
{
  "cluster": "eks-b",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
},
{
  "cluster": "dev",
  "asgname": "eks-12345",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
},
{
  "cluster": "dev",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
},
{
  "cluster": "eks-a",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
},
{
  "cluster": "eks-a",
  "asgname": "eks-1323",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
},
]

jq validation:

$ cat state.json  | jq empty
parse error: Expected another array element at line 44, column 1

I have attempted multiple options, mostly with sed, but have not met the specific requirement to only remove the last occurrence of }, Any advice is appreciated.

CodePudding user response：

Let your script just produce a stream of objects by not printing the brackets and the commas.

echo "${JSON_STRING}" >> state.json

state.json should then contain a stream of objects:

{
  "cluster": "eks-b",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
}
{
  "cluster": "eks-b",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
}
{
  "cluster": "dev",
  "asgname": "eks-12345",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
}
{
  "cluster": "dev",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
}
{
  "cluster": "eks-a",
  "asgname": "eks-1234",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
}
{
  "cluster": "eks-a",
  "asgname": "eks-1323",
  "minSize": "0",
  "DesiredCapacity": "0",
  "MaxSize": "0"
}

Now you can read in that stream with jq, using the -s option to convert it into a proper array.

jq -s . state.json

[
  {
    "cluster": "eks-b",
    "asgname": "eks-1234",
    "minSize": "0",
    "DesiredCapacity": "0",
    "MaxSize": "0"
  },
  {
    "cluster": "eks-b",
    "asgname": "eks-1234",
    "minSize": "0",
    "DesiredCapacity": "0",
    "MaxSize": "0"
  },
  {
    "cluster": "dev",
    "asgname": "eks-12345",
    "minSize": "0",
    "DesiredCapacity": "0",
    "MaxSize": "0"
  },
  {
    "cluster": "dev",
    "asgname": "eks-1234",
    "minSize": "0",
    "DesiredCapacity": "0",
    "MaxSize": "0"
  },
  {
    "cluster": "eks-a",
    "asgname": "eks-1234",
    "minSize": "0",
    "DesiredCapacity": "0",
    "MaxSize": "0"
  },
  {
    "cluster": "eks-a",
    "asgname": "eks-1323",
    "minSize": "0",
    "DesiredCapacity": "0",
    "MaxSize": "0"
  }
]

Demo for the jq -s part

CodePudding user response：

If you don't require the final result to be pretty-printed, you could save the expense of the call to jq -s by deleting the superfluous comma, e.g.

sed '$s/,$//'

or if you like:

sed -i.bak '$s/,$//'

Even better, you can avoid handling the terminating ']' in the same step, e.g. using '$s/,$/]/'