I originally had an issue with a JSON (let's call it "raw_data.json") that looked like this:

[
  {
    "metadata": {
      "project": [
        "someProject1",
        "someProject2"
      ],
      "table_name": [
        "someTable1",
        "someTable2"
      ],
      "sys_insertdatetime": "someDate",
      "sys_sourcesystem": "someSourceSystem"
    },
    "data": {
      "field1": 63533712,
      "field2": "",
      "field3": "hello",
      "field4": "other",
      "field5": 2022,
      "field6": "0",
      "field7": "0",
      "field8": "0",
      "field9": "0",
      "field10": "0",
      "field11": "0"
    }
  },
  {
    "metadata": {
      "project": [
        "someProject1",
        "someProject2"
      ],
      "table_name": [
        "someTable1",
        "someTable2"
      ],
      "sys_insertdatetime": "someDate",
      "sys_sourcesystem": "someSourceSystem"
    },
    "data": {
      "field1": 63533713,
      "field2": "Y2JjLTIwMjItdzA1LXRyZi1vZmZyZXMtcmVuZm9ydC1jaGVxdWllci13MDU=",
      "field3": "A0AVB",
      "field4": "other",
      "field5": "HJlbmZvcnQgY2hlcXVpZXIgVzA1",
      "field6": "",
      "field7": "02/02/2022",
      "field8": "14/02/2022",
      "field9": "Ticket"
    }
  }
]

The dictionaries were pretty big and usually spanned over several lines on my monitor. That was an issue because I needed to encode the whole thing in Base64 and send the result with a HTTP POST method. But the encoding tended to wrap lines by adding newlines characters, which caused my POST method to fail.

I fortunately found a question with the following solution:

export DATA=$(cat 'raw_data.json')
export ENCODED_DATA=$(echo "$DATA" | jq -c . | base64 -w 0)

Problem is, my JSON has now changed to this:

{
  "request_id": 1234,
  "data_to_be_encoded": [
    {
      "metadata": {
        "project": [
          "someProject1",
          "someProject2"
        ],
        "table_name": [
          "someTable1",
          "someTable2"
        ],
        "sys_insertdatetime": "someDate",
        "sys_sourcesystem": "someSourceSystem"
      },
      "data": {
        "field1": 63533712,
        "field2": "",
        "field3": "hello",
        "field4": "other",
        "field5": 2022,
        "field6": "0",
        "field7": "0",
        "field8": "0",
        "field9": "0",
        "field10": "0",
        "field11": "0"
      }
    },
    {
      "metadata": {
        "project": [
          "someProject1",
          "someProject2"
        ],
        "table_name": [
          "someTable1",
          "someTable2"
        ],
        "sys_insertdatetime": "someDate",
        "sys_sourcesystem": "someSourceSystem"
      },
      "data": {
        "field1": 63533713,
        "field2": "Y2JjLTIwMjItdzA1LXRyZi1vZmZyZXMtcmVuZm9ydC1jaGVxdWllci13MDU=",
        "field3": "A0AVB",
        "field4": "other",
        "field5": "HJlbmZvcnQgY2hlcXVpZXIgVzA1",
        "field6": "",
        "field7": "02/02/2022",
        "field8": "14/02/2022",
        "field9": "Ticket"
      }
    }
  ]
}

Basically I needed to keep the request_id key-value pair as is, while what was inside the data_to_be_encoded key had to be encoded. Again, another post offered a nice solution:

export DATA=$(cat 'raw_data.json')
export ENCODED_DATA=$(echo "$DATA" | jq '.data_to_be_encoded |= @base64')

Except for the fact that this solution adds line wrapping and I haven't found a way to disable that feature like I managed to do with the first solution.

I did try this:

export ENCODED_DATA=$(echo "$DATA" | jq -c .data_to_be_encoded | base64 -w 0)

But it only returns the value inside the data_to_be_encoded key and not the whole JSON. So I'm back to square one.

How can I get the best of both worlds? In other words, how can I disable the line wrapping while at the same time encoding a specific part of my JSON?

CodePudding user response：

Use jq's @base64 builtin, rather than doing the conversion outside because that way you can either convert all or nothing.

jq '.data_to_be_encoded |= @base64'

{
  "request_id": 1234,
  "data_to_be_encoded": "W3sibWV0YWRhdGEiOnsicHJvamVjdCI6WyJzb21lUHJvamVjdDEiLCJzb21lUHJvamVjdDIiXSwidGFibGVfbmFtZSI6WyJzb21lVGFibGUxIiwic29tZVRhYmxlMiJdLCJzeXNfaW5zZXJ0ZGF0ZXRpbWUiOiJzb21lRGF0ZSIsInN5c19zb3VyY2VzeXN0ZW0iOiJzb21lU291cmNlU3lzdGVtIn0sImRhdGEiOnsiZmllbGQxIjo2MzUzMzcxMiwiZmllbGQyIjoiIiwiZmllbGQzIjoiaGVsbG8iLCJmaWVsZDQiOiJvdGhlciIsImZpZWxkNSI6MjAyMiwiZmllbGQ2IjoiMCIsImZpZWxkNyI6IjAiLCJmaWVsZDgiOiIwIiwiZmllbGQ5IjoiMCIsImZpZWxkMTAiOiIwIiwiZmllbGQxMSI6IjAifX0seyJtZXRhZGF0YSI6eyJwcm9qZWN0IjpbInNvbWVQcm9qZWN0MSIsInNvbWVQcm9qZWN0MiJdLCJ0YWJsZV9uYW1lIjpbInNvbWVUYWJsZTEiLCJzb21lVGFibGUyIl0sInN5c19pbnNlcnRkYXRldGltZSI6InNvbWVEYXRlIiwic3lzX3NvdXJjZXN5c3RlbSI6InNvbWVTb3VyY2VTeXN0ZW0ifSwiZGF0YSI6eyJmaWVsZDEiOjYzNTMzNzEzLCJmaWVsZDIiOiJZMkpqTFRJd01qSXRkekExTFhSeVppMXZabVp5WlhNdGNtVnVabTl5ZEMxamFHVnhkV2xsY2kxM01EVT0iLCJmaWVsZDMiOiJBMEFWQiIsImZpZWxkNCI6Im90aGVyIiwiZmllbGQ1IjoiSEpsYm1admNuUWdZMmhsY1hWcFpYSWdWekExIiwiZmllbGQ2IjoiIiwiZmllbGQ3IjoiMDIvMDIvMjAyMiIsImZpZWxkOCI6IjE0LzAyLzIwMjIiLCJmaWVsZDkiOiJUaWNrZXQifX1d"
}

Demo

If you need all in one line, use the -c option as before:

jq -c '.data_to_be_encoded |= @base64'

{"request_id":1234,"data_to_be_encoded":"W3sibWV0YWRhdGEiOnsicHJvamVjdCI6WyJzb21lUHJvamVjdDEiLCJzb21lUHJvamVjdDIiXSwidGFibGVfbmFtZSI6WyJzb21lVGFibGUxIiwic29tZVRhYmxlMiJdLCJzeXNfaW5zZXJ0ZGF0ZXRpbWUiOiJzb21lRGF0ZSIsInN5c19zb3VyY2VzeXN0ZW0iOiJzb21lU291cmNlU3lzdGVtIn0sImRhdGEiOnsiZmllbGQxIjo2MzUzMzcxMiwiZmllbGQyIjoiIiwiZmllbGQzIjoiaGVsbG8iLCJmaWVsZDQiOiJvdGhlciIsImZpZWxkNSI6MjAyMiwiZmllbGQ2IjoiMCIsImZpZWxkNyI6IjAiLCJmaWVsZDgiOiIwIiwiZmllbGQ5IjoiMCIsImZpZWxkMTAiOiIwIiwiZmllbGQxMSI6IjAifX0seyJtZXRhZGF0YSI6eyJwcm9qZWN0IjpbInNvbWVQcm9qZWN0MSIsInNvbWVQcm9qZWN0MiJdLCJ0YWJsZV9uYW1lIjpbInNvbWVUYWJsZTEiLCJzb21lVGFibGUyIl0sInN5c19pbnNlcnRkYXRldGltZSI6InNvbWVEYXRlIiwic3lzX3NvdXJjZXN5c3RlbSI6InNvbWVTb3VyY2VTeXN0ZW0ifSwiZGF0YSI6eyJmaWVsZDEiOjYzNTMzNzEzLCJmaWVsZDIiOiJZMkpqTFRJd01qSXRkekExTFhSeVppMXZabVp5WlhNdGNtVnVabTl5ZEMxamFHVnhkV2xsY2kxM01EVT0iLCJmaWVsZDMiOiJBMEFWQiIsImZpZWxkNCI6Im90aGVyIiwiZmllbGQ1IjoiSEpsYm1admNuUWdZMmhsY1hWcFpYSWdWekExIiwiZmllbGQ2IjoiIiwiZmllbGQ3IjoiMDIvMDIvMjAyMiIsImZpZWxkOCI6IjE0LzAyLzIwMjIiLCJmaWVsZDkiOiJUaWNrZXQifX1d"}

Demo