Use yq to update array value-CodePudding

I have two yaml files I am using. The first yaml file looks like this:

spring.yml

spring:
  cloud:
    gateway:
      routes:
      - id: someid
        uri: someUri
        predicates:
        - Path=/somePath
        filters:
        - RewritePath=/someOtherPath

I have another file that just contains routes and looks like this:

routes.yml

routes:
  - id: someid
    uri: someOtherUri
    predicates:
    - Path=/somePath
    filters:
    - RewritePath=/someNewPath

My goal is to update the route in the first file with the value of the route in the second file. Note that the first file in reality will have many routes but for demonstration purposes I am only showing the first in this example. I have the following script which loops through to update the routes as necessary when the id's match:

#!/bin/sh
OVERRIDE_ROUTE_IDS=$(yq eval '.routes.[].id' routes.yml)
GENERATED_ROUTE_IDS=$(yq eval '.spring.cloud.gateway.routes.[].id' spring.yml)

SAVEIFS=$IFS   # Save current IFS (Internal Field Separator)
IFS=$'\n'      # Change IFS to newline char
OVERRIDE_ROUTE_IDS=($OVERRIDE_ROUTE_IDS) # split the `OVERRIDE_ROUTE_IDS` string into an array by the same name
GENERATED_ROUTE_IDS=($GENERATED_ROUTE_IDS) # split the `GENERATED_ROUTE_IDS` string into an array by the same name
IFS=$SAVEIFS   # Restore original IFS

for (( i=0; i<${#OVERRIDE_ROUTE_IDS[@]}; i   ))
do
   if [[ "${GENERATED_ROUTE_IDS[*]}" =~ "${OVERRIDE_ROUTE_IDS[$i]}" ]]
   then
      echo "route ID ${OVERRIDE_ROUTE_IDS[$i]} exists in generated routes"
      for (( j=0; j<${#GENERATED_ROUTE_IDS[@]}; j   ))
      do
         if [[ "${GENERATED_ROUTE_IDS[$j]}" == "${OVERRIDE_ROUTE_IDS[$i]}" ]]
         then
            echo "index of route ${GENERATED_ROUTE_IDS[$j]} is $j"
            echo "$i"
            ROUTE_TO_USE=$(yq eval ".routes.[$i]" routes.yml)
            $(yq ".spring.cloud.gateway.routes.[$j] = $ROUTE_TO_USE" spring.yml)
         fi
      done
   else
      echo "no match so add to top of routes"
   fi
done

My assumption is this command should update spring.yml file with the new route in place of the one that was identified with the same id:

$(yq ".spring.cloud.gateway.routes.[$j] = $ROUTE_TO_USE" application.yml)

But I am getting the following error

Error: Parsing expression: Lexer error: could not match text starting at 1:37 failing at 1:39 unmatched text: "id"

I'm stumped on this and not sure what I'm doing wrong at this point. For reference I am using yq version 4.17.2.

CodePudding user response：

Be aware that yq does not emit a data structure, it emits a string. $ROUTE_TO_USE would be, for example,

id: someid
uri: someOtherUri
predicates:
  - Path=/somePath
filters:
  - RewritePath=/someNewPath

This is YAML source. Pasting this into the following yq command leads to invalid syntax; yq's expression syntax is not literal YAML. This is what the error tries to tell you.

What you want to do is to process both inputs in a single yq command:

yq ea "select(fi==0).spring.cloud.gateway.routes.[$j] = "`
     `"select(fi==1).routes.[$i] | select(fi==0)" spring.yml routes.yml

ea is shorthand for eval-all which you need for processing multiple input files at the same time. fi is a shorthand for fileIndex, which is used to select the appropriate file. Piping the result to | select(fi==0) ensures that only the first (modified) file is written out. I split the long string into multiple lines using backticks for readability.

CodePudding user response：

I ended up getting a solution from the creator of yq. The solution below is what I used:

yq ea '
  (select(fi==0) | .spring.cloud.gateway.routes.[].id) as $currentIds |
  (select(fi==1) | [.routes.[] | select( [$currentIds != .id] | all )] ) as $newRoutes |
  ( $newRoutes   .spring.cloud.gateway.routes   .routes) as $routesToMerge |

  (
    (($routesToMerge | .[] | {.id: .}) as $item ireduce ({}; . * $item )) as $uniqueMap
    | ( $uniqueMap  | to_entries | .[]) as $item ireduce([]; .   $item.value)
  ) as $mergedArray
  | select(fi == 0) | .spring.cloud.gateway.routes = $mergedArray  

' spring.yml routes.yml

This matches on id. If there is a match it uses the value of what's in routes.yml. If there is no match it add it the top top of the routes.