How to run curl commands from gitlab-ci CI/CD on gitlab/github?-CodePudding

I have this .gitlab-ci.yml file content like:

.install-pre-reqs: &install-pre-reqs
  image: ubuntu:20.04
  before_script:
    - apt-get update -y
    - apt-get upgrade -y
    - apt-get install -y zip unzip curl fastjar
    - chmod  x deploy.sh

test:
    <<: *install-pre-reqs
    stage: test
    script:
        - echo 'test'
        - ./deploy.sh
        - echo "content"
        - exit 0

And the deploy.sh script which contains a curl command:

#!/bin/sh
curl -u "admin:admin" -X POST "http://localhost:9998/rest/admin/system/restart"

I want to be able to run the curl command through CI/CD. When I run the command directly with curl on my local, it is working without issues. However, the configured CI/CD pipeline triggers this error messages:

Executing "step_script" stage of the job script
00:32
Using docker image sha256:3bc6e9f30f51d2bbf9307fc9d0bdfc30caa38cf4e3b05a714230f9a9b3381d84 for ubuntu:20.04 with digest ubuntu@sha256:af5efa9c28de78b754777af9b4d850112cad01899a5d37d2617bb94dc63a49aa ...
$ apt-get update -y
Err:1 http://security.ubuntu.com/ubuntu focal-security InRelease
  Could not connect to security.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.38), connection timed out Could not connect to security.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.39), connection timed out
Err:2 http://archive.ubuntu.com/ubuntu focal InRelease
  Could not connect to archive.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.38), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out
Err:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease
  Unable to connect to archive.ubuntu.com:http:
Err:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease
  Unable to connect to archive.ubuntu.com:http:
Reading package lists...
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease  Could not connect to archive.ubuntu.com:80 (91.189.91.39), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.38), connection timed out Could not connect to archive.ubuntu.com:80 (185.125.190.39), connection timed out
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease  Unable to connect to archive.ubuntu.com:http:
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease  Unable to connect to archive.ubuntu.com:http:
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease  Could not connect to security.ubuntu.com:80 (185.125.190.39), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.38), connection timed out Could not connect to security.ubuntu.com:80 (185.125.190.36), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.39), connection timed out
W: Some index files failed to download. They have been ignored, or old ones used instead.
$ apt-get upgrade -y
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
$ apt-get install -y zip unzip curl fastjar
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package zip
E: Unable to locate package unzip
E: Unable to locate package curl
E: Unable to locate package fastjar
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

How could I solve these issues and make the curl command run on the gitlab environment without issues?

CodePudding user response：

Are you using gitlab.com or self hosted? It seems that the pipeline runner is behind a proxy and dont't have access to install your packages:

http://security.ubuntu.com/ubuntu Could not connect to security.ubuntu.com:80

If self hosted, you probably have to ask permission for the machine where the runner is hosted to be able to download your packages.
If you are using gitlab.com runners, maybe it was some downtime in the archive.ubuntu.com servers? I created a copy of your gitlab-ci.yml and it worked fine.

Not related to your question, but check this answer about how to improve your pipeline setup: https://stackoverflow.com/a/66012365/9697259

TL;DR: It would be better to create a custom docker image with your dependencies already installed to save time during the pipeline execution.

CodePudding user response：

Another solution if you're using a self hosted GitLab server with restricted access to the internet: use an existing/official/maintained Docker image with required tools (provided your runners allow using Docker images from Docker Hub).

This way you won't bother with building/maintaining a home-made Docker image. Another good point is that you won't apt-get update, apt-get upgrade, apt-get install over and over again (it will save time). Last but not least: a size-optimized image (such as Alpine) will be ways smaller than a base Linux image such as Ubuntu 20, also shortening your pipeline execution.

Personally for curl jq stuff I like using dwdraju/alpine-curl-jq (<5MB)

Your .gitlab-ci.yml would become:

test:
  image:
    name: "dwdraju/alpine-curl-jq"
    entrypoint: [""]
    stage: test
    script:
        - echo 'test'
        - ./deploy.sh
        - echo "content"
        - exit 0