bash text processing to remove ascii and get unique lines from the result-CodePudding

In linux, below command

terraform providers

output the result as below

.
├── provider[xxx.com/edu/xxxvenafi] 1.2.0
├── provider[registry.terraform.io/hashicorp/kubernetes] 2.3.2
├── provider[xxx.com/edu/xxxsmaas] 1.0.1
├── provider[registry.terraform.io/hashicorp/aws]
├── module.standard_deployment
│   ├── provider[xxx.com/edu/xxxsmaas] 1.0.1
│   ├── provider[xxx.com/edu/xxxvenafi] 1.2.0
│   ├── provider[registry.terraform.io/hashicorp/kubernetes]
│   └── provider[registry.terraform.io/hashicorp/local]
└── module.standand_ingress
    ├── provider[registry.terraform.io/hashicorp/kubernetes]
    ├── provider[xxx.com/edu/xxxsmaas] 1.0.1
    ├── provider[xxx.com/edu/xxxvenafi] 1.2.0
    └── provider[registry.terraform.io/hashicorp/aws]

Providers required by state:

    provider[xxx.com/edu/xxxsmaas]

    provider[xxx.com/edu/xxxvenafi]

    provider[registry.terraform.io/hashicorp/aws]

    provider[registry.terraform.io/hashicorp/kubernetes]

What is the best way to remove those tree structure from the output? The end goal is to list only unique lines as below,

provider[xxx.com/edu/xxxvenafi] 1.2.0    
provider[xxx.com/edu/xxxsmaas] 1.0.1

CodePudding user response：

Using sed and sort, you can try this

$ sed -E 's/.*(provider.*)/\1/g;/^[a-z]/!d' input_file | sort -u
provider[registry.terraform.io/hashicorp/aws]
provider[registry.terraform.io/hashicorp/kubernetes]
provider[registry.terraform.io/hashicorp/kubernetes] 2.3.2
provider[registry.terraform.io/hashicorp/local]
provider[xxx.com/edu/xxxsmaas] 1.0.1
provider[xxx.com/edu/xxxvenafi] 1.2.0

This may not be the most efficient however.

CodePudding user response：

With awk:

terrraform providers |
  awk 'BEGIN {FS="provider"}
    /xxx.com/ && NF==2 {printf("%s%s\n", FS, $2)}' | 
  sort -u

or avoiding the call to sort:

terrraform providers |
 awk 'BEGIN {FS="provider"}
   /xxx.com/ && NF==2 {a[$2]}
   END {
     for (key in a) {
       printf("provider%s\n", key)
     }
   }'

CodePudding user response：

$ awk '{sub(/\r$/,"")} /[0-9]$/ && sub(/[^[:alpha:]] /,"") && !seen[$0]  ' file
provider[xxx.com/edu/xxxvenafi] 1.2.0
provider[registry.terraform.io/hashicorp/kubernetes] 2.3.2
provider[xxx.com/edu/xxxsmaas] 1.0.1

or if you really only want lines ending in 1.0.1 or 1.2.0 as you said in a comment:

$ awk '{sub(/\r$/,"")} / 1\.((0\.1)|(2\.0))$/ && sub(/[^[:alpha:]] /,"") && !seen[$0]  ' file
provider[xxx.com/edu/xxxvenafi] 1.2.0
provider[xxx.com/edu/xxxsmaas] 1.0.1