We are getting the output from an API call in the below list of dictionaries format,

[
{
        "Key": "/builder-deployer/test2/services/serviceA/ip",
        "Val": "10.1.2.1"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceA/port",
        "Val": "2"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceA/url",
        "Val": "serviceA.abc.com"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceA/username",
        "Val": "jenkins"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceB/ip",
        "Val": "10.1.2.2"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceB/port",
        "Val": "3"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceB/url",
        "Val": "serviceB.abc.com"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceB/username",
        "Val": "jenkins"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceC/ip",
        "Val": "10.1.2.2"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceC/port",
        "Val": "4"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceC/url",
        "Val": "serviceC.abc.com"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceC/username",
        "Val": "jenkins"
    }
]

I have a requirement to fetch the unique username and IP combination in the form of username@ip from the list to install some dependencies. i.e. in the above list of dictionaries, we have 2 unique username@ip combinations. So I need to install packages on [email protected], [email protected] machines respectively.

I have the below code in a python script that does the same but most of our code is in Bash, we want to convert this also to Bash. Any help would be very helpful

ip=[]
for username in outputs:
    for ip in outputs:
        if username['Key'].split('/')[-2] == ip['Key'].split('/')[-2] and username['Key'].split('/')[-1] in ['username'] and ip['Key'].split('/')[-1] in ['ip']:
            ip.append(username['Val'] '@' ip['Val'])
print(set(ip))

CodePudding user response：

jq

$ FILE="file.json"
$ paste -d@ \
        <(jq -r '.[]|select(.Key|contains("/username")).Val' "$FILE") \
        <(jq -r '.[]|select(.Key|contains("/ip")).Val' "$FILE") | \
        sort -u

[email protected]
[email protected]    

awk

awk -F\" '/\/ip/{getline;ip=$4}/username/{getline;$0=$4"@"ip;print|"sort -u"}' "$FILE"

[email protected]
[email protected]

CodePudding user response：

The suggestion in the comments to leave your solution in python (per Gordon Davisson) would be my preference as well. But, if you really have to have a bash solution, the following is one option. Bear in mind it will be fragile if your API JSON format changes over time. It is also not terribly efficient as it relies on reading the source JSON multiple occasions. Further, both grep and sort are also utilized:

#!/bin/bash

input_json="${1:-/tmp/bar/t.json}"

while read -r line ; do
    search_str=$(grep -oB 1 "${line}" "${input_json}" \
     | awk 'NR==1 {gsub(/^"/,"", $2); \
     gsub(/ip",$/,"username", $2); print $2}')
    
    user=$(grep -oA 1 "${search_str}" "${input_json}" | \
     awk 'NR==2 {gsub(/"/,"", $2); print $2}')
    
    echo "${user}@${line}"

# get unique ip addresses from source json
done < <(grep -Eo "([0-9]{1,3}[\.]){3}[0-9]{1,3}" "${input_json}" | sort -u)

Sample data:

[
{
        "Key": "/builder-deployer/test2/services/serviceA/ip",
        "Val": "10.1.2.1"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceA/port",
        "Val": "2"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceA/url",
        "Val": "serviceA.abc.com"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceA/username",
        "Val": "jenkins"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceB/ip",
        "Val": "10.1.2.2"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceB/port",
        "Val": "3"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceB/url",
        "Val": "serviceB.abc.com"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceB/username",
        "Val": "jenkins"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceC/ip",
        "Val": "10.1.2.2"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceC/port",
        "Val": "4"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceC/url",
        "Val": "serviceC.abc.com"
    },
    {
        "Key": "/builder-deployer/test2/services/serviceC/username",
        "Val": "jenkins"
    }
]

Sample output:

$ ./t.sh
[email protected]
[email protected]