How to convert a multiple-line variable containing JSON to single line in shell?-CodePudding

I have below variable in shell

{
  "type": "service_account",
  "project_id": "projectid234",
  "private_key_id": "aasdadsxzce5",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5 xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5 mXuN45vwqh6QvG4lw/Hi7EJhtAn FQy7 yOQYrw3l\nQ2CpxDotT PT2OuQ6LVbc/F SblPlrK3B 8aEMo57PZ gnwMcQ7 ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "1234242342341",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"
}

Like below:

export var1='{
      "type": "service_account",
      "project_id": "projectid234",
      "private_key_id": "aasdadsxzce5",
      "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5 xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5 mXuN45vwqh6QvG4lw/Hi7EJhtAn FQy7 yOQYrw3l\nQ2CpxDotT PT2OuQ6LVbc/F SblPlrK3B 8aEMo57PZ gnwMcQ7 ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n",
      "client_email": "[email protected]",
      "client_id": "1234242342341",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://oauth2.googleapis.com/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"
    }'

Now I want to convert it to a single line, what should I do? I tried various solutions but they didn't work. for instance I used below code which I found but after converting to base64 and decoding it, it just return the first line. would you mind help me?

$((echo $var1 | tr -d '\n') | base64)

CodePudding user response：

Since you're dealing with JSON, I'd recommend a tool like jq, which provides us with the --compact-output option and is capable of encoding to Base64 using @base64

To single-line output

jq --compact-output '' <<< "$test"

To Base64 single-line output

jq --compact-output  '@base64' <<< "$test"

To Base64 single-line output, and decoding with bash

jq --compact-output --raw-output '@base64' <<< "$test" | base64 --decode

Note the --raw-output to remove the ""'s.

Local shell example of above lines:

$ test='{
  "type": "service_account",
  "project_id": "projectid234",
  "private_key_id": "aasdadsxzce5",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5 xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5 mXuN45vwqh6QvG4lw/Hi7EJhtAn FQy7 yOQYrw3l\nQ2CpxDotT PT2OuQ6LVbc/F SblPlrK3B 8aEMo57PZ gnwMcQ7 ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "1234242342341",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"
}'
$
$ jq --compact-output '' <<< "$test"
{"type":"service_account","project_id":"projectid234","private_key_id":"aasdadsxzce5","private_key":"-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5 xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5 mXuN45vwqh6QvG4lw/Hi7EJhtAn FQy7 yOQYrw3l\nQ2CpxDotT PT2OuQ6LVbc/F SblPlrK3B 8aEMo57PZ gnwMcQ7 ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n","client_email":"[email protected]","client_id":"1234242342341","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"}
$
$
$ jq --compact-output '@base64' <<< "$test"
"eyJ0eXBlIjoic2VydmljZV9hY2NvdW50IiwicHJvamVjdF9pZCI6InByb2plY3RpZDIzNCIsInByaXZhdGVfa2V5X2lkIjoiYWFzZGFkc3h6Y2U1IiwicHJpdmF0ZV9rZXkiOiItLS0tLUJFR0lOIFBSSVZBVEUgS0VZLS0tLS1cbk1JSUV2Z0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktnd2dnU2tBZ0VBQW9JQkFRQ2lXakE1K3hGNmdzR0Ncbk9rbklMOURPQ1FwVjJMUlhZUEVLbDUrbVh1TjQ1dndxaDZRdkc0bHcvSGk3RUpodEFuK0ZReTcreU9RWXJ3M2xcblEyQ3B4RG90VCtQVDJPdVE2TFZiYy9GK1NibFBsckszQis4YUVNbzU3UForZ253TWNRNytvZlBuekM2MzV1VVBcbnBPRzBpZE1US1xuLS0tLS1FTkQgUFJJVkFURSBLRVktLS0tLVxuIiwiY2xpZW50X2VtYWlsIjoiY2xpbmV0bWFpbEBhc2Rhc2QzNDM1LmlhbS5nc2VydmljZWFjY291bnQuY29tIiwiY2xpZW50X2lkIjoiMTIzNDI0MjM0MjM0MSIsImF1dGhfdXJpIjoiaHR0cHM6Ly9hY2NvdW50cy5nb29nbGUuY29tL28vb2F1dGgyL2F1dGgiLCJ0b2tlbl91cmkiOiJodHRwczovL29hdXRoMi5nb29nbGVhcGlzLmNvbS90b2tlbiIsImF1dGhfcHJvdmlkZXJfeDUwOV9jZXJ0X3VybCI6Imh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL29hdXRoMi92MS9jZXJ0cyIsImNsaWVudF94NTA5X2NlcnRfdXJsIjoiaHR0cHM6Ly93d3cuZ29vZ2xlYXBpcy5jb20vcm9ib3QvdjEvbWV0YWRhdGEveDUwOS9hc3NhZGphc2hkLmlhbS5nc2VydmljZWFjY291bnQuY29tIn0="
$
$
$ jq --compact-output --raw-output '@base64' <<< "$test" | base64 --decode
{"type":"service_account","project_id":"projectid234","private_key_id":"aasdadsxzce5","private_key":"-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5 xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5 mXuN45vwqh6QvG4lw/Hi7EJhtAn FQy7 yOQYrw3l\nQ2CpxDotT PT2OuQ6LVbc/F SblPlrK3B 8aEMo57PZ gnwMcQ7 ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n","client_email":"[email protected]","client_id":"1234242342341","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"}%
$
$

CodePudding user response：

tl;dr version -

$: echo "${var1//[[:space:]]/}" # // means global replacement 
{"type":"service_account","project_id":"projectid234","private_key_id":"aasdadsxzce5","private_key":"-----BEGINPRIVATEKEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5 xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5 mXuN45vwqh6QvG4lw/Hi7EJhtAn FQy7 yOQYrw3l\nQ2CpxDotT PT2OuQ6LVbc/F SblPlrK3B 8aEMo57PZ gnwMcQ7 ofPnzC635uUP\npOG0idMTK\n-----ENDPRIVATEKEY-----\n","client_email":"[email protected]","client_id":"1234242342341","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"}

Then to break it down and out a bit... Just as a quick pass, you really should be able to just do this with built-in parameter expansion in bash.

The simplest:

$: echo "$var1" # this should present it exactly as you saved it
{
      "type": "service_account",
      "project_id": "projectid234",
      "private_key_id": "aasdadsxzce5",
      "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5 xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5 mXuN45vwqh6QvG4lw/Hi7EJhtAn FQy7 yOQYrw3l\nQ2CpxDotT PT2OuQ6LVbc/F SblPlrK3B 8aEMo57PZ gnwMcQ7 ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n",
      "client_email": "[email protected]",
      "client_id": "1234242342341",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://oauth2.googleapis.com/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"
    }

$: echo "${var1//$'\n'/}" # this should strip out the newlines
{      "type": "service_account",      "project_id": "projectid234",      "private_key_id": "aasdadsxzce5",      "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5 xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5 mXuN45vwqh6QvG4lw/Hi7EJhtAn FQy7 yOQYrw3l\nQ2CpxDotT PT2OuQ6LVbc/F SblPlrK3B 8aEMo57PZ gnwMcQ7 ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n",      "client_email": "[email protected]",      "client_id": "1234242342341",      "auth_uri": "https://accounts.google.com/o/oauth2/auth",      "token_uri": "https://oauth2.googleapis.com/token",      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"    }

Assuming you also want to squeeze out extraneous whitespace with a more complex pattern, you might be tempted to use a regex...which won't work, because the parameter parsing actually usesw globbing, but regular expressions per se... but try shopt for extended globbing -

$: shopt -s extglob # c.f. https://mywiki.wooledge.org/glob#extglob
$: echo "${var1//$'\n' ( )/}" # remove a newline followed by one or more spaces
{"type": "service_account","project_id": "projectid234","private_key_id": "aasdadsxzce5","private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5 xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5 mXuN45vwqh6QvG4lw/Hi7EJhtAn FQy7 yOQYrw3l\nQ2CpxDotT PT2OuQ6LVbc/F SblPlrK3B 8aEMo57PZ gnwMcQ7 ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n","client_email": "[email protected]","client_id": "1234242342341","auth_uri": "https://accounts.google.com/o/oauth2/auth","token_uri": "https://oauth2.googleapis.com/token","auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"}

Since you seem to have no whitespaces embedded in your data (that I noticed), you could take out the requirement for a newline in front of them, and just remove ALL spaces to compact it even more.

$: echo "${var1//*($'\n') ( )/}"
{"type":"service_account","project_id":"projectid234","private_key_id":"aasdadsxzce5","private_key":"-----BEGINPRIVATEKEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5 xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5 mXuN45vwqh6QvG4lw/Hi7EJhtAn FQy7 yOQYrw3l\nQ2CpxDotT PT2OuQ6LVbc/F SblPlrK3B 8aEMo57PZ gnwMcQ7 ofPnzC635uUP\npOG0idMTK\n-----ENDPRIVATEKEY-----\n","client_email":"[email protected]","client_id":"1234242342341","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"}

So...in this case, with no embedded spaces, you can achieve the best result without extended globbing at all, by just using POSIX character classes to remove all whitespace, as at the top. :)

$: echo "${var1//[[:space:]]/}" # // means global replacement
{"type":"service_account","project_id":"projectid234","private_key_id":"aasdadsxzce5","private_key":"-----BEGINPRIVATEKEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5 xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5 mXuN45vwqh6QvG4lw/Hi7EJhtAn FQy7 yOQYrw3l\nQ2CpxDotT PT2OuQ6LVbc/F SblPlrK3B 8aEMo57PZ gnwMcQ7 ofPnzC635uUP\npOG0idMTK\n-----ENDPRIVATEKEY-----\n","client_email":"[email protected]","client_id":"1234242342341","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"}

CodePudding user response：

Not sure I understand the use of base64 but a simple tr should work, eg:

$ var2=$(echo ${var1} | tr -d '\n')              # ${var1} is not wrapped in double quotes so repetitive white space is squeezed to single spaces

$ echo "${var2}"
{ "type": "service_account", "project_id": "projectid234", "private_key_id": "aasdadsxzce5", "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5 xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5 mXuN45vwqh6QvG4lw/Hi7EJhtAn FQy7 yOQYrw3l\nQ2CpxDotT PT2OuQ6LVbc/F SblPlrK3B 8aEMo57PZ gnwMcQ7 ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n", "client_email": "[email protected]", "client_id": "1234242342341", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com" }

Or as William Pursell pointed out in comments, don't wrap ${var1} in quotes and you can also eliminate the od call:

$ var2=$(echo ${var1})                           # the unquoted ${var1} also removes linefeeds so no need for tr

$ echo "${var2}"
{ "type": "service_account", "project_id": "projectid234", "private_key_id": "aasdadsxzce5", "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5 xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5 mXuN45vwqh6QvG4lw/Hi7EJhtAn FQy7 yOQYrw3l\nQ2CpxDotT PT2OuQ6LVbc/F SblPlrK3B 8aEMo57PZ gnwMcQ7 ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n", "client_email": "[email protected]", "client_id": "1234242342341", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com" }

Or if you wish to maintain all the extra white space you can wrap ${var1} in double quotes before passing to tr, eg:

$ var2=$(echo "${var1}" | tr -d '\n')            # ${var1} is wrapped in double quotes to maintain all white space

$ echo "${var2}"
{      "type": "service_account",      "project_id": "projectid234",      "private_key_id": "aasdadsxzce5",      "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5 xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5 mXuN45vwqh6QvG4lw/Hi7EJhtAn FQy7 yOQYrw3l\nQ2CpxDotT PT2OuQ6LVbc/F SblPlrK3B 8aEMo57PZ gnwMcQ7 ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n",      "client_email": "[email protected]",      "client_id": "1234242342341",      "auth_uri": "https://accounts.google.com/o/oauth2/auth",      "token_uri": "https://oauth2.googleapis.com/token",      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"    }