compare two variables ignoring specified string-CodePudding

I am trying to compare two list of IP address space set as separate variables but one of the lists has additional IP range that I'd like to skip in comparison. How can I skip it? grep apparently can skip the whole line containing the string, tr removes all character occurrences. I am downloading the fresh IP ranges:

wget https://www.cloudflare.com/ips-v4 && wget https://www.cloudflare.com/ips-v6

then set it as first variable:

L1=$(awk '{printf fmt,$1}' fmt="%s\n" ips-v4 ips-v6 | paste -sd, -)

then from the existing YAML file I am setting second variable:

L2=$(yq -r .controller.config.proxy-real-ip-cidr yaml.yaml)

The L2 has additional string/IP range that I'd like to skip in comparison: 1.2.3.0/32 Later I am going to send an email if there is a difference between both (except the ignored range)

L1 output:

echo "$L1"                                                         
173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32

L2 output:

echo "$L2"
173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/12,172.64.0.0/13,131.0.72.0/22,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32,1.2.3.0/32

CodePudding user response：

$ diff <(tr ',' '\n' <<<"$L1"|sort) <(tr ',' '\n' <<<"$L2"|sort) | \
  awk '/^< /{print "L1 ONLY >>>",$2} 
       /^> /{if($2 != "1.2.3.0/32") print "L2 ONLY >>>",$2}'
L1 ONLY >>> 104.16.0.0/13
L1 ONLY >>> 104.24.0.0/14
L2 ONLY >>> 104.16.0.0/12

awk '
  BEGIN{
    exclude_arr["1.2.3.0/32"]
  }
  FNR==NR{ l1[$0]; next }
  { l2[$0] }
  END{
    for (i in l1)
        if (i in l2 == 0 ) print "L1 ONLY >>>", i
    for (i in l2)
        if (i in l1 == 0 && i in exclude_arr == 0) print "L2 ONLY >>>", i
 }
 ' <(tr ',' '\n' <<<"$L1") <(tr ',' '\n' <<<"$L2")

L1 ONLY >>> 104.16.0.0/13
L1 ONLY >>> 104.24.0.0/14
L2 ONLY >>> 104.16.0.0/12

CodePudding user response：

You can use sed (stream editor) for that task:

╰─$ foo="asdftestASDF"      

╰─$ echo "$foo"                             
asdftestASDF

╰─$ echo "$foo" | sed 's/test/bar/'
asdfbarASDF

╰─$ echo "$L2" | sed -E 's/\,1\.2\.3\.0\/32//'
173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/12,172.64.0.0/13,131.0.72.0/22,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2a06:98c0::/29,2c0f:f248::/32

Note that I am escaping dots . and slash / to avoid them being interpreted as a special symbol in regex or sed. You could go without escaping the dots, but as . in regex matches any character, this could lead to unexpected results.