How to capture output of curl piped with grep into a variable in a shell script-CodePudding

I realise there are quite a few posts that show how to capture output of curl or grep, but haven't been able to find any post where I can capture the value of a group in the grep's regular expression.

I need to capture output of the following command in a variable inside a shell script.

I am working on API calls of Qlik tool. The way the APIs work, is that you first have to call the login operation, which returns a session key and then use this session key in the subsequent API calls, like this

https://help.qlik.com/en-US/enterprise-manager/November2021/Content/EnterpriseManager/EnterpriseManager_APIGuide/CurlAPI/api_login.htm

cURL example
Active Directory request
curl -i -k --header "Authorization: Basic cWFAcWE6cWE=" https://computer.network.net/attunityenterprisemanager/api/v1/login

Response
HTTP/1.1 200 OK

Content-Length: 0

Content-Type: text/html

Server: Microsoft-HTTPAPI/2.0

EnterpriseManager.APISessionID: J3cKzWIbi_w6Fr1G-tO03Q

Date: Mon, 26 Dec 2016 17:02:01 GMT

And then use the EnterpriseManager.APISessionID

https://help.qlik.com/en-US/enterprise-manager/November2021/Content/EnterpriseManager/EnterpriseManager_APIGuide/CurlAPI/api_aem_getserver.htm

So, I am trying to get this EnterpriseManager.APISessionID value in a variable, so I can use it in the script to call other operations.

If I run the below command on the command line it prints the value on the terminal,

curl -i -k --header "Authorization: Basic mybase64idpwvalue" https://qlik-qem-dev.abc.com/attunityenterprisemanager/api/v1/login | grep '(EnterpriseManager.APISessionID: )(.*)' | echo "${BASH_REMATCH[2]}"

This is the output on command line

eu_3qrEWJFAz_4hL7bOIvA
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

but if I put it in a .sh file and execute it, then the variable value is empty

API_SESSION_ID=$(curl -i -k --header "Authorization: Basic mybase64idpwvalue" https://qlik-qem-dev.abc.com/attunityenterprisemanager/api/v1/login | grep 'EnterpriseManager.APISessionID: ')
API_SESSION_ID2=$(curl -i -k --header "Authorization: Basic mybase64idpwvalue" https://qlik-qem-dev.abc.com/attunityenterprisemanager/api/v1/login | grep '(EnterpriseManager.APISessionID: )(.*)')
#API_SESSION_ID=$(curl -i -k --header "Authorization: Basic mybase64idpwvalue" https://qlik-qem-dev.abc.com/attunityenterprisemanager/api/v1/login) | grep '(EnterpriseManager.APISessionID: )(.*)' |  echo "${BASH_REMATCH[2]}"
echo "API_SESSION_ID : ${API_SESSION_ID}"
echo "API_SESSION_ID2 : ${API_SESSION_ID2}"

I tried some other combinations of the command in the script, but am unable to capture the value in a variable. It is always empty

How can I get that populated?

Thanks

CodePudding user response：

First, I don't know where eu_3qrEWJFAz_4hL7bOIvA comes from in your example. I'll assume you meant J3cKzWIbi_w6Fr1G-tO03Q.

Second, your command line output doesn't make sense, unless you played around with the shell and some global state remains. When I run it, I get a empty output, which is what I expect.

Third, BASH_REMATCH is a bash variable that captures the result of regex matching from bash operators ([[ string =~ regex ]]). It's not aware of grep, perl, sed, or any other programs using regex.
That's why when I run your command I get no output - BASH_REMATCH is empty.

Last, your grep expression uses unquoted parentheses, and you didn't pass -E, so it looks for parentheses in the text, and it wouldn't find them. So even if BATCH_REMATCH was aware of grep, grep would still return an empty result.

All that said, sed does support capture:

API_SESSION_ID="$( \
    curl -i -k \
        --header "Authorization: Basic mybase64idpwvalue" \
        https://qlik-qem-dev.abc.com/attunityenterprisemanager/api/v1/login \
        | sed -n 's/\(EnterpriseManager.APISessionID: \)\(.*\)/\2/p' \
)"
echo "API_SESSION_ID : ${API_SESSION_ID}"

That \2 represents the second capture group. But you don't actually need the first:

API_SESSION_ID="$( \
    curl -i -k \
        --header "Authorization: Basic mybase64idpwvalue" \
        https://qlik-qem-dev.abc.com/attunityenterprisemanager/api/v1/login \
        | sed -n 's/EnterpriseManager.APISessionID: \(.*\)/\1/p' \
)"
echo "API_SESSION_ID : ${API_SESSION_ID}"

CodePudding user response：

BASH_REMATCH isn't related to grep. It must be called after regex matching.

GNU grep (perl positive look behind):

echo "$example_header" |
grep -Po '(?<=^EnterpriseManager\.APISessionID: ).*'

Pure bash:

[[ $example_header =~ (^|$'\n')(EnterpriseManager\.APISessionID: )([^$'\n']*) ]]
echo "${BASH_REMATCH[3]}"

Both print:

J3cKzWIbi_w6Fr1G-tO03Q

For input:

example_header=\
'Response
HTTP/1.1 200 OK

Content-Length: 0

Content-Type: text/html

Server: Microsoft-HTTPAPI/2.0

EnterpriseManager.APISessionID: J3cKzWIbi_w6Fr1G-tO03Q

Date: Mon, 26 Dec 2016 17:02:01 GMT'

# Or: example_header=$(curl -i -k --header "Authorization: Basic cWFAcWE6cWE=" https://computer.network.net/attunityenterprisemanager/api/v1/login)

Don't forget to escape . and to match the start of the line (to avoid matching NotEnterpriseManager...).