CURL script containing AWK fails over SSH to print Docker Image Digest-CodePudding

The following curl script fails over ssh,

    #!/bin/bash
    reg='dockreg:5000'
    image='mubu6'
    itag='v6'
    auth='-u user:pass'
    accept=(
    -H "Accept: application/vnd.docker.distribution.manifest.v2 json"
    )
    
    ssh -tt [email protected] "
    
    echo get digest; read
    curl ${auth} -vsk \
    -X DELETE \
      "https://${reg}/v2/${image}/manifests/$(
        curl ${auth} \
        -vk "${accept[@]}" \
        https://${reg}/v2/${image}/manifests/${itag} 2>&1 |\
    grep docker-content-digest | awk '{print $3}' |\
    tr -d $'\r'
    )"
    "

without printing the digest after manifests on the first output line below, while throwing a 404 error on the last.

DELETE /v2/mubu6/manifests/ HTTP/2
> Host: dockreg:5000
> authorization: Basic YWxleGFuZGVyOnNvZmlhbm9z
> user-agent: curl/7.68.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 404

It runs successfully on the node though,

echo get digest; read
curl ${auth} -vsk \
-X DELETE \
  "https://${reg}/v2/${image}/manifests/$(
    curl ${auth} \
    -vk "${accept[@]}" \
    https://${reg}/v2/${image}/manifests/${itag} 2>&1 |\
grep docker-content-digest | awk '{print $3}' |\
tr -d $'\r'
)"

printing the digest after manifests on the first line and a HTTP/2 202 exit code on the last:

DELETE /v2/mubu6/manifests/sha256:0aa2280cc066ef4f8279122fc9f76d15e96a8bfa642a54dadbf8c9985f3de747 HTTP/2
> Host: dockreg:5000
> authorization: Basic YWxleGFuZGVyOnNvZmlhbm9z
> user-agent: curl/7.68.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 202

The failure is probably related to awk as it already needs tr -d $'\r' in order to pass the line feed correctly to curl. Furthermore, I already know that awk does not work out of the box over ssh from only running successfully the following, after escaping $2 in awk:

for x in ${NODE_IPS[@]}; do
ssh -tt root@$x "
dpkg -l | grep '^rc' | awk '{print \$2}' | \
xargs -p dpkg --purge
echo -e "continue to next node \c"; read
"

Unfortunately trying the same solution with my curl over ssh script throws errors:

awk: cmd. line:1: {print \$3}
awk: cmd. line:1:        ^ backslash not last character on line
awk: cmd. line:1: {print \$3}
awk: cmd. line:1:        ^ syntax error

I 've also tried quite a few different quoting permutations all over the script to no avail. I am very new to all this and any help would be highly appreciated.

CodePudding user response：

You have three sets of double quotes nested inside each other. They are causing havoc with the bash parsing. If you want the quotes inside transported to the far end of the SSH tunnel you need to escape them with backslashes...

ssh is probably only seeing the content inside the first set of quotes:

ssh -tt [email protected] "
    
    echo get digest; read
    curl ${auth} -vsk \
    -X DELETE \
      "

Bash has 2 types of quotes: single quotes and double quotes.

single quotes will cause bash to NOT interpret any escapes, variables, etc. inside the string.

double quotes will cause bash to do full substitution of escapes and variables.

You can place single quotes inside double quotes and variables will still be replaced:

DEMOVAR="just testing"
echo "This is a 'test of $DEMOVAR' to show single quotes have no effect here"

# RESULT:
This is a 'test of just testing' to show single quotes did not affect substitution

But it does not work the same when placing double quotes inside single quotes:

echo 'This is a "test of $DEMOVAR" to show that single quotes prevent substitution'

# RESULT: 
This is a "test of $DEMOVAR" to show that single quotes prevent substitution

To make it more complicated, you can 'escape' doublequotes inside doublequotes, but you cannot escape single quotes inside single quotes:

echo "This set of \"Double quotes\" will show"

But single quotes will concatenate:

echo 'Single quotes \'will not escape\' inside single quotes'

The above line will open a single quote and then close it right after the first backslash-quote. The following text: will not escape' inside single quotes is not inside any quotes and so the second backslash-quote is escaped by bash. Finally the quote at the end of the line OPENS a new QUOTE block so hitting enter will have it awaiting another quote to close the block.

Confused??? It took me years to understand this nuance

To make your code even more difficult, you are wrapping it inside an SSH which transmits the stuff from the first doublequote to the second doublequote, but the rest of the lines are not really inside double quotes. The question is WHERE are the bash substitutions done? You are trying a $(curl xxxxx) to get the digest but I can't really tell if you want that on the local machine or on the remote machine.

I would suggest you break up your commands into multiple SSH commands to get rid of the nesting:

#!/bin/bash
reg='dockreg:5000'
image='mubu6'
itag='v6'
auth='-u user:pass'
# Note I put double quotes inside single quotes
accept='-H "Accept: application/vnd.docker.distribution.manifest.v2 json"'

# NOT sure what this is:
echo get digest; read

# Get the DIGEST. Note that I got rid of the 'grep'
# as awk can also filter, and I moved the 'tr' before the 'awk'
# I am only doing the 'curl' on the remote machine, and  
# the entire output is transmitted back.  The filters ('tr' and 'awk' )
# are done locally and the DIGEST variable is set locally.

DIGEST=$(ssh [email protected] \
    "curl ${auth} -vk ${accept} https://${reg}/v2/${image}/manifests/${itag} 2>&1" |\
    tr -d '\r' |awk '/docker-content-digest/ {print $3}');

# Now call the DELETE with the preset DIGEST.
ssh [email protected] \
    "curl ${auth} -vsk -X DELETE https://${reg}/v2/${image}/manifests/${DIGEST}"