Tests in Gitab CI pipeline marked as passed when 2 tests cases fail-CodePudding

We have a project managed in Gitlab, with CI pipeline for builds and tests (pytest, Google tests). Two or three of our test cases in Google tests fail. But Gitlab consider that the test stage is successful. Is it because the success percentage is more than 90% (an arbitrary value) ? Is there a way to make the stage (and thus the complete pipeline) fail if we don't get 100% of success ?

Here is a screenshot of the pipeline summary:

Here is the yml script of the stage:

test_unit_test:
  stage: test
  needs: ["build", "build_unit_test"]
  image: $DOCKER_IMAGE
  rules:
    - if: '$CI_PIPELINE_SOURCE != "merge_request_event"'
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  script: |
    ZIPNAME=`cat _VERSION_.txt`
    ./scripts/gitlab-ci/stage-unittests.sh test_unit_test_report.xml $ZIPNAME
  artifacts:
    reports:
      junit: test_unit_test_report.xml
    expire_in: 1 week

Thank you for any help. Regards.

CodePudding user response：

Gitlab CI/CD jobs don't care what the script areas are doing (so they don't look at, for example, test pass percentages). The only they things use to determine if a job passed or failed are exit codes and the allow_failure keyword.

After each command in the before_script, script, and after_script sections are executed, the Gitlab Runner checks the exit code of the command to see if it is 0. If it is non-zero, the command is considered a failure, and if the allow_failure keyword is not set to true for the job, the job fails.

So, for your job, even though the tests are failing, somehow the script is existing with exit code 0, meaning the command itself finished successfully. The command in this case is:

ZIPNAME=$(cat _VERSION_.txt)
./scripts/gitlab-ci/stage-unittests.sh test_unit_test_report.xml $ZIPNAME

NOTE: I replaced your backticks '`' with the $(command) syntax explained here which does the same thing (execute this command) but has some advantages over '`command`' including nesting and easier use in markdown where '`' indicates code formatting.

So, since you are calling a script (./scripts/gitlab-ci/stage-unittests.sh) to run your tests, that script itself is finishing successfully, so the job finishes successfully. Take a look at that script to see if you can tell why it finishes successfully even though the tests fail.