I've got a build script that is running in Gitlab and generates some files that are used later in the build process.

The reason is Gitlab pipeline fails and the failure is not reproduced locally. Is there a way to troubleshoot the failure?

As far as I know Gitlab pipelines are running in Docker containers.

Is there a way to get the docker image of the failed Gitlab pipeline to analyze it locally (e.g. take a look at the generated files)?

CodePudding user response：

When the job container exits, it is removed automatically, so this would not be feasible to do.

However, you might have a few other options to debug your job:

Interactive web session

If you are using self-hosted runners, the best way to do this would probably be with a interative web session. That would allow you to have an interactive shell session inside the container. (be aware, you may have to edit the job to sleep for some time in order to keep the container alive long enough to inspect it)

Artifact files

If you're not using self-hosted runners, another option would be to artifact the generated files on failure:

artifacts:
  when: on_failure
  paths:
  - path/to/generated-files/**/*

You can then download the artifacts and debug them locally.

Use the job script to debug

Yet another option would be to add debugging output to the job itself.

script:
  - generate-files
  # this is just an example, you can make this more helpful,
  # depending on what information you need for debugging
  - cat path/to/generated-files/*

Because debugging output may be noisy, you can consider using collapsible sections to collapse debug output by default.

script:
  - generate-files
  - echo "Starting debug section"

  # https://docs.gitlab.com/ee/ci/jobs/index.html#custom-collapsible-sections
  - echo -e "\e[0Ksection_start:`date  %s`:debugging[collapsed=true]\r\e[0KGenerated File Output"
  - cat path/to/generated-files/*
  - echo -e "\e[0Ksection_end:`date  %s`:debugging\r\e[0K"

Use the gitlab-runner locally

You can run jobs locally with, more or less, the same behavior of the GitLab runner by installing the gitlab runner locally and using the gitlab-runner exec command.

In this case, you could run your job locally and then docker exec into your job:

1. In your local repo, start the job by running the gitlab-runner exec command, providing the name of the job
2. In another shell, run docker ps to find the container ID of the job started by gitlab-runner exec
3. exec into the container using its ID: docker exec -it <CONTAINER_ID> /bin/bash (assuming bash is available in your image)