I've got a jq command that works when running directly from the shell or from within a shell script, but when I try to add variable expansion, I get jq errors for unexpected format or invalid characters. My goal is to have a quick and easy way to update some json configuration.

Here's a simplified example.

The format of the json I'm modifying:

{
  "pets": {
      "some-new-pet": {
          "PetInfo": {
             "name": "my-brand-new-pet",
             "toys": [
                 "toy1-postfix",
                 "toy2-postfix",
                 "toy3-postfix"
              ]
          }
      }  
   }
}

The jq without variable expansion:

cat myfile.json | jq '.pets."some-new-pet"  = {PetInfo: {name: "my-brand-new-pet"}, toys: ["toy1", "toy2", "toy3"]}}' 

The above runs fine, and adds the new pets.some-new-pet entry to my json.

Below is what I'm trying to do with variable expansion that fails.

jq_args = "'.pets.\"${PET}\"  = {PetInfo: {name: \"${NAME}\"}, toys: [\"${toy1}-postfix\", \"${toy2}-postfix\", \"${toy3}-postfix\"]}}'"
cat myfile.json | jq $jq_args

The error message I get with the above:

jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1: '.pets."some-new-pet"

My file is formatted as utf-8 and uses LF line endings.

CodePudding user response：

It would be cleaner and less error prone to format the string using printf

PET='dog'
NAME='sam'
toy1="t1"
toy2="t2"
toy3="t3"

jq_args=$(printf '.pets."%s"  = {PetInfo: {name: "%s"}, toys: ["%s-postfix", "%s-postfix", "%s-postfix"]}}' "${PET}" "${NAME}" "${toy1}" "${toy2}" "${toy3}")
echo "$jq_args"

Result:

.pets."dog"  = {PetInfo: {name: "sam"}, toys: ["t1-postfix", "t2-postfix", "t3-postfix"]}

Additionally, redundant quoting could be avoided by quoting the arg on this command

cat myfile.json | jq "$jq_args"

CodePudding user response：

1. Fix your jq code by removing extra } at end

2. Fix bash jq call:

   Add cotes "..." around your $jq_args

   so don't use singles '...' in your jq_args definition

3. Use printf with -v option to define jq_args:

   printf -v jq_args "...format..." value1 value2 ...

So your code became:

PET="some-new-pet"
NAME="my-brand-new-pet"
toy1="toy1"
toy2="toy2"
toy3="toy3"
format='.pets."%s"  = {PetInfo: {name: "%s"}, toys: ["%s", "%s", "%s"]}'
printf -v jq_args "${format}" "${PET}" "${NAME}" "${toy1}" "${toy2}" "${toy3}"
cat myfile.json | jq "$jq_args"

Output:

{
  "pets": {
    "some-new-pet": {
      "PetInfo": {
        "name": "my-brand-new-pet"
      },
      "toys": [
        "toy1",
        "toy2",
        "toy3"
      ]
    }
  }
}

Notes:

• When you define your format, you put it into simple cotes '...'. It's really better to format JSON (or XML) without back-slashes (\\) before each double cotes (")
• Use printf -v variable_name. It's more readable than var_name=$(printf ...)

CodePudding user response：

I do not recommend constructing a jq filter using variable expansion or printf. It will work for simple cases but will fail if the string contains double quotes, backslashes or control-codes, as they have special meanings inside a JSON string. As an alternative to using printf, jq has a way to pass in variables directly via the command-line, avoiding all these issues.

pet='some-second-pet'
name='my-even-newer'
toy1=toy1
toy2=toy2
toy3=toy3
jq \
    --arg pet "$pet" \
    --arg name "$name" \
    --arg toy1 "$toy1" \
    --arg toy2 "$toy2" \
    --arg toy3 "$toy3" \
    '.pets[$pet]  = {
        PetInfo: {name: $name},
        toys: ["\($toy1)-postfix", "\($toy2)-postfix", "\($toy3)-postfix"]
    }' \
    myfile.json

Output:

{
  "pets": {
    "some-new-pet": {
      "PetInfo": {
        "name": "my-brand-new-pet",
        "toys": [
          "toy1-postfix",
          "toy2-postfix",
          "toy3-postfix"
        ]
      }
    },
    "some-second-pet": {
      "PetInfo": {
        "name": "my-even-newer-pet"
      },
      "toys": [
        "toy1-postfix",
        "toy2-postfix",
        "toy3-postfix"
      ]
    }
  }
}

CodePudding user response：

By constructing the jq filter ("code") using outer bash variables ("data") you may run into escaping issues, which could eventually break or even divert your filter. (see https://en.wikipedia.org/wiki/Code_injection)

Instead, use mechanisms by jq to introduce external data through variables (parameter --arg):

jq --arg pet "${PET}" \
  --arg name "${NAME}" \
  --arg toy1 "${toy1}-postfix" \
  --arg toy2 "${toy2}-postfix" \
  --arg toy3 "${toy3}-postfix" \
  '
    
    .pets[$pet]  = {PetInfo: {$name, toys: [$toy1,$toy2,$toy3]}}
    
  ' myfile.json

If you have an unknown number of variables to include, check out jq's --args parameter (note the additional s)