Bash JSON compare two list and delete id-CodePudding

I have a JSON endpoint which I can fetch value with curl and yml local file. I want to get the difference and delete it with id of name present on JSON endpoint.

JSON's endpoint

[
  {
    "hosts": [
      "server1"
    ],
    "id": "qz9o847b-f07c-49d1-b1fa-e5ed0b2f0519",
    "name": "V1_toto_a"
  },
  {
    "hosts": [
      "server2"
    ],
    "id": "a6aa847b-f07c-49d1-b1fa-e5ed0b2f0519",
    "name": "V1_tata_b"
  },
  {
    "hosts": [
      "server3"
    ],
    "id": "a6d9ee7b-f07c-49d1-b1fa-e5ed0b2f0519",
    "name": "V1_titi_c"
  }
]

files.yml

---
instance:
  toto:
    name: "toto"
  tata:
    name: "tata"

Between JSON's endpoint and local file, I want to delete it with id of tata, because it is the difference between the sources.

declare -a arr=(_a _b _c)
ar=$(cat files.yml | grep name | cut -d '"' -f2 | tr "\n" " ")
fileItemArray=($ar)
ARR_PRE=("${fileItemArray[@]/#/V1_}")
for i in "${arr[@]}"; do local_var =("${ARR_PRE[@]/%/$i}"); done
remote_var=$(curl -sX GET "XXXX" | jq -r '.[].name | @sh' | tr -d \'\")
diff_=$(echo ${local_var[@]} ${remote_var[@]} | tr ' ' '\n' | sort | uniq -u)

output = titi

the code works, but I want to delete the titi with id dynamically

curl -X DELETE "XXXX" $id_titi

I am trying to delete with bash script, but I have no idea to continue...

CodePudding user response：

Your endpoint is not proper JSON as it has

commas after the .name field but no following field
no commas between the elements of the top-level array

If this is not just a typo from pasting your example into this question, then you'd need to address this first before proceeding. This is how it should look like:

[
  {
    "hosts": [
      "server1"
    ],
    "id": "qz9o847b-f07c-49d1-b1fa-e5ed0b2f0519",
    "name": "toto"
  },
  {
    "hosts": [
      "server2"
    ],
    "id": "a6aa847b-f07c-49d1-b1fa-e5ed0b2f0519",
    "name": "tata"
  },
  {
    "hosts": [
      "server3"
    ],
    "id": "a6d9ee7b-f07c-49d1-b1fa-e5ed0b2f0519",
    "name": "titi"
  }
]

If your endpoint is proper JSON, try the following. It extracts the names from your .yml file (just as you do - there are plenty of more efficient and less error-prone ways but I'm trying to adapt your approach as much as possible) but instead of a Bash array generates a JSON array using jq which for Bash is a simple string. For your curl output it's basically the same thing, extracting a (JSON) array of names into a Bash string. Note that in both cases I use quotes <var>="$(…)" to capture strings that may include spaces (although I also use the -c option for jq to compact it's output to a single line). For the difference between the two, everything is taken over by jq as it can easily be fed with the JSON arrays as variables, perform the subtraction and output in your preferred format:

fromyml="$(cat files.yml | grep name | cut -d '"' -f2 | jq -Rnc '[inputs]')"
fromcurl="$(curl -sX GET "XXXX" | jq -c 'map(.name)')"
diff="$(jq -nr --argjson fromyml "$fromyml" --argjson fromcurl "$fromcurl" '
  $fromcurl - $fromyml | .[]
')"

The Bash variable diff now contains a list of names only present in the curl output ($fromcurl - $fromyml), one per line (if, other than in your example, there happens to be more than one). If the curl output had duplicates, they will still be included (use $fromcurl - $fromyml | unique | .[] to get rid of them):

titi

As you can see, this solution has three calls to jq. I'll leave it to you to further reduce that number as it fits your general workflow (basically, it can be put together into one).

CodePudding user response：

Getting the output of a program into a variable can be done using read.

perl -M5.010 -MYAML -MJSON::PP -e'
   sub get_next_file { local $/; "".<> }
   my %filter = map { $_->{name} => 1 } values %{ Load(get_next_file)->{instance} };
   say for grep !$filter{$_}, map $_->{name}, @{ decode_json(get_next_file) };
' b.yaml a.json |
while IFS= read -r id; do
   printf 'Do something with %s\n' "$id"
done

Output:

Do something with titi

I used Perl here because what you had was no way to parse a YAML file. The snippet requires having installed the YAML Perl module.