where is the docker image stored in gitlab ci?-CodePudding

I have build a docker image successfully and tag it as testdock:latest ($CI_REGISTRY_IMAGE:latest) the $CI_REGISTRY variable is kept in GitLab project variable.

I have another stage , to start scanning the testdock image by using Trivy: the process is just stuck without progress. I am guessing is that the image cannot be found or something wrong with the docker environment in GitLab.

   Where is the `docker image (testdock)` stored?

this is the command that I used for Trivy to scan the testdock image:

$ TRIVY_INSECURE=true trivy --skip-update --output "$CI_PROJECT_DIR/scanning-report.json"  $CI_REGISTRY_IMAGE:latest

the yml:

build:
  stage: build
  image: $CI_REGISTRY/devops/docker:latest
  services:
    - $CI_REGISTRY/devops/docker:dind-nx1.0
  #tags:
  #  - docker
  variables:
    # No need to clone the repo, we exclusively work on artifacts.  See
    # https://docs.gitlab.com/ee/ci/runners/README.html#git-strategy
    TRIVY_USERNAME: "$CI_REGISTRY_USER"
    TRIVY_PASSWORD: "$CI_REGISTRY_PASSWORD"
    TRIVY_AUTH_URL: "$CI_REGISTRY"
    FULL_IMAGE_NAME: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    # Tell docker CLI how to talk to Docker daemon.
    DOCKER_HOST: tcp://localhost:2375/
    # Use the overlayfs driver for improved performance.
    DOCKER_DRIVER: overlay2
    # Disable TLS since we're running inside local network.
    DOCKER_TLS_CERTDIR: ""
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker build -t $FULL_IMAGE_NAME  .
   # - docker push $CI_REGISTRY_IMAGE:latest

security_scan:
  stage: test
  image: 
    name: $CI_REGISTRY/devops/trivy/trivy:0.20.1
    entrypoint: [""]
  services:
    - $CI_REGISTRY/devops/docker:dind-nx1.0
  #tags:
   # - docker
  variables:
    # No need to clone the repo, we exclusively work on artifacts.  See
    # https://docs.gitlab.com/ee/ci/runners/README.html#git-strategy
  #  GIT_STRATEGY: none
    TRIVY_USERNAME: "$CI_REGISTRY_USER"
    TRIVY_PASSWORD: "$CI_REGISTRY_PASSWORD"
    TRIVY_AUTH_URL: "$CI_REGISTRY"
    FULL_IMAGE_NAME: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    # Tell docker CLI how to talk to Docker daemon.
    DOCKER_HOST: tcp://localhost:2375/
    # Use the overlayfs driver for improved performance.
    DOCKER_DRIVER: overlay2
    # Disable TLS since we're running inside local network.
    DOCKER_TLS_CERTDIR: ""
  before_script:
    - git config --global http.sslVerify false
    - git clone $CI_REPOSITORY_URL
    - echo "the project directory is - $CI_PROJECT_DIR"
    - echo "the CI_REGISTRY_IMAGE variable is - $CI_REGISTRY_IMAGE"
    - echo "the full image name is - $FULL_IMAGE_NAME"
    - ls -la
    - trivy -h | grep cache
    - mkdir -p /root/.cache/trivy/db
    - ls -la
    - cp "eval-trivy-2/trivy-offline.db.tgz" "/root/.cache/trivy/db"
    - cd /root/.cache/trivy/db
    - tar xvf trivy-offline.db.tgz
    - ls -la
  script:
    - trivy --version
    - time trivy image --clear-cache
    # running 1 hr and stopped.
    #- TRIVY_INSECURE=true trivy --skip-update $CI_REGISTRY_IMAGE:latest
    #- TRIVY_INSECURE=true trivy --skip-update -f json -o scanning-report.json $CI_REGISTRY/devops/aquasec/trivy:0.16.0
    - TRIVY_INSECURE=true trivy --skip-update -o "$CI_PROJECT_DIR/scanning-report.json" $FULL_IMAGE_NAME
    #keep loading by using testdock:latest
    #- TRIVY_INSECURE=true trivy --skip-update -o "$CI_PROJECT_DIR/scanning-report.json"  testdock:latest
   # - TRIVY_INSECURE=true trivy --skip-update --exit-code 1 --severity CRITICAL $CI_REGISTRY/devops/aquasec/trivy:0.16.0
  artifacts:
    when:                          always
    reports:
      container_scanning:          scanning-report.json

CodePudding user response：

All jobs are running isolated. Therefore jobA normally does not know what jobB produced as long as you do not tell the job specifically to pass things on to the next job with the artifacts directive.

In your case you build your image in your job, but if you did not push it - it will be just like any throw away data and lost at the next stage. The easiest way is to push it to a docker registry and use it from there. eg. a common practice is to tag it with the commit SHA instead of latest. This way you can ensure you are always hitting the right image.

CodePudding user response：

final gitlan-ci.yml which works well now:

variables:
  # Tell docker CLI how to talk to Docker daemon.
  DOCKER_HOST: tcp://localhost:2375/
  # Use the overlayfs driver for improved performance.
  DOCKER_DRIVER: overlay2
  # Disable TLS since we're running inside local network.
  DOCKER_TLS_CERTDIR: ""


services:
  - $CI_REGISTRY/devops/docker:dind-nx1.0

stages:
  - build
  - test

#include:
  # Trivy integration with GitLab Container Scanning
 # - remote: "https://github.com/aquasecurity/trivy/raw/master/contrib/Trivy.gitlab-ci.yml"

build:
  image: $CI_REGISTRY/devops/docker:latest
  stage: build
  variables:
    IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  script:
    - docker info
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    - docker build -t $IMAGE .
    - docker tag $IMAGE $CI_REGISTRY/$IMAGE
    - docker push $CI_REGISTRY/$IMAGE 

Trivy_container_scanning:
  stage: test
  image:
    name: $CI_REGISTRY/devops/trivy/trivy:0.20.1
  variables:
    # Override the GIT_STRATEGY variable in your `.gitlab-ci.yml` file and set it to `fetch` if you want to provide a `clair-whitelist.yml`
    # file. See https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html#overriding-the-container-scanning-template
    # for details
    GIT_STRATEGY: none
    IMAGE: "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
  allow_failure: true
  before_script:
    - trivy image --reset
    - git config --global http.sslVerify false
    - git clone $CI_REPOSITORY_URL
    - echo "the project directory is - $CI_PROJECT_DIR"
    - echo "the registry image is - $CI_REGISTRY_IMAGE"
    - ls -la
    - trivy -h | grep cache
    - mkdir -p /root/.cache/trivy/db
    - ls -la
    - cp "eval-trivy-4/trivy-offline.db.tgz" "/root/.cache/trivy/db"
    - cd /root/.cache/trivy/db
    - tar xvf trivy-offline.db.tgz
    - ls -la
    #- export TRIVY_VERSION=${TRIVY_VERSION:-v0.19.2}
    #- apk add --no-cache curl docker-cli
    #- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    #- curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin ${TRIVY_VERSION}
    #- curl -sSL -o /tmp/trivy-gitlab.tpl https://github.com/aquasecurity/trivy/raw/${TRIVY_VERSION}/contrib/gitlab.tpl
  script:
    - TRIVY_INSECURE=true trivy image --skip-update -f json -o "$CI_PROJECT_DIR/gl-container-scanning-report.json" $CI_REGISTRY/$IMAGE
  #unable to write results: failed to initialize template writer: error retrieving template from path: open /tmp/trivy-gitlab.tpl: no such file or directory
   # - TRIVY_INSECURE=true trivy image --skip-update --format template --template "@/tmp/trivy-gitlab.tpl" -o gl-container-scanning-report.json $CI_REGISTRY/$IMAGE
    #scan error
    #- trivy --skip-update --format template --template "@/tmp/trivy-gitlab.tpl" -o gl-container-scanning-report.json $CI_REGISTRY/$IMAGE
    #- trivy --exit-code 0 --cache-dir .trivycache/ --no-progress --format template --template "@/tmp/trivy-gitlab.tpl" -o gl-container-scanning-report.json $IMAGE
 # cache:
  #  paths:
 #     - .trivycache/
  artifacts:
    reports:
      container_scanning: gl-container-scanning-report.json

reference and modified for my env

https://gitlab.com/aquasecurity/trivy-ci-test/-/blob/master/.gitlab-ci.yml