I tried with jq to prase some JSON output inside my GitLab CI pipeline so I can extract needed information. I've tried many different ways but I can't get the desired information out of the target node because it has special characters and when I get to that node the pipeline fails in each case. This is the current state of my pipeline, any suggestion or help would be appreciated.

This is a problematic job:

get results (dev branch):
  stage: Results of scanning image
  variables:
    RESULTS: ""
    STATUS: ""
    SEVERITY: ""
  image: alpine
  only:
    refs:
      - dev
  allow_failure: true
  before_script:
    - apk update && apk upgrade
    - apk --no-cache add curl
    - apk add jq
  script:
    - 'RESULTS=$(curl -H "Authorization: Basic `echo -n ${HARBOR_USER}:${HARBOR_PASSWORD} | base64`" -X GET "https://url.to.registry/api/v2.0/projects/project/repositories/repo-name/artifacts/latest?page=1&page_size=10&with_tag=true&with_label=true&with_scan_overview=true&with_signature=true&with_immutable_status=true")'
    - echo $RESULTS
    - RESULTS=$RESULTS | tr 'application/vnd.scanner.adapter.vuln.report.harbor json; ' 'myobject'
    - echo $RESULTS
    - "STATUS=$RESULTS | jq '.scan_overview .myobjectversion=1.0 .scan_status'"
    - "SEVERITY=$RESULTS | jq '.scan_overview .myobjectversion=1.0 .severity'"
    - echo "Printing the results of the image scanning process on Harbor registry:"
    - echo "status of scan:$STATUS"
    - echo "severity of scan:$SEVERITY"
    - echo "For more information of scan results please visit Harbor registry!"
  tags:
    - dev
    - docker

This is JSON output that I get from curl command:

{
   "addition_links":{
      "build_history":{
         "absolute":false,
         "href":"..."
      },
      "vulnerabilities":{
         "absolute":false,
         "href":"...."
      }
   },
   "digest":"sha256:bcd665be2b7c6725b410029db385d7c6c71a9ce557427cbd0f54d01a9",
   "extra_attrs":{
      "architecture":"amd64",
      "author":null,
      "created":"2021-10-22T10:28:46.058276455Z",
      "os":"linux"
   },
   "icon":"sha256:0048162a053ee7518615bef084403614f8bca43b40ae2e762e11e06",
   "id":362,
   "labels":null,
   "manifest_media_type":"application/vnd.docker.distribution.manifest.v2 json",
   "media_type":"application/vnd.docker.container.image.v1 json",
   "project_id":3,
   "pull_time":"2021-10-22T10:28:55.305Z",
   "push_time":"2021-10-22T10:28:49.341Z",
   "references":null,
   "repository_id":12,
   "scan_overview":{
      "application/vnd.scanner.adapter.vuln.report.harbor json; version=1.0":{
         "complete_percent":100,
         "duration":8,
         "end_time":"2021-10-22T10:28:57.356Z",
         "report_id":"e83854eb-2304-4c58-85c9-a3e0fd9067a8",
         "scan_status":"Success",
         "severity":"Critical",
         "start_time":"2021-10-22T10:28:49.827Z",
         "summary":{
            "summary":{
               "Critical":7,
               "High":47,
               "Low":18,
               "Medium":47
            },
            "total":119
         }
      }
   }
}

My initial idea was to using jq extarct scan_status and severity with this command:

RESULTS=$RESULTS | jq '.scan_overview .application/vnd.scanner.adapter.vuln.report.harbor json; version=1.0 .scan_status'

after running that command I got this error:

jq: error: syntax error, unexpected ';', expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.scan_overview .application/vnd.scanner.adapter.vuln.report.harbor json; version=1.0 .scan_status                                                                       
jq: 1 compile error

now I am trying text replacment but that doesn't work also.

How should I proceed in this case ?

CodePudding user response：

Use square brackets and double quotes around the problematic key:

jq '.scan_overview["application/vnd.scanner.adapter.vuln.report.harbor json; version=1.0"].scan_status'

CodePudding user response：

Wrap the key in question inside barckets and quotes like so:

.scan_overview["application/vnd.scanner.adapter.vuln.report.harbor json; version=1.0"].scan_status

Demo

CodePudding user response：

If it is okay for you to ignore the name of the problematic key, then you can substitute it with empty brackets:

RESULTS=$RESULTS | jq '.scan_overview [] .scan_status'

This is a bit sloppy and may match more than you would like, but in your narrow example, it will successfully pull out the value of .scan_status.

The reason this works, in this case, is because the application/vnd... property is the only property in the scan_overview object.

A more confident match would be achievable by using more quotes:

RESULTS=$RESULTS | jq '.scan_overview ."application/vnd.scanner.adapter.vuln.report.harbor json; version=1.0" .scan_status'