Our registered Gitlab-runner (on Kubernetes) was working fine, after upgrading the version of Gitlab, it can't clone the projects anymore! Does anyone have any idea about this issue?

Here is the log of the issue:

Running with gitlab-runner 14.9.0 (d1f69508)
  on gitlab-runner-dev K5KVWdx-
Preparing the "kubernetes" executor
30:00
Using Kubernetes namespace: cicd
Using Kubernetes executor with image <docker-registry>:kuber_development ...
Using attach strategy to execute scripts...
Preparing environment
30:07
Waiting for pod cicd/runner-k5kvwdx--project-1227-concurrent-02kqgq to be running, status is Pending
Waiting for pod cicd/runner-k5kvwdx--project-1227-concurrent-02kqgq to be running, status is Pending
    ContainersNotReady: "containers with unready status: [build helper]"
    ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-k5kvwdx--project-1227-concurrent-02kqgq via gitlab-runner-85776bd9c6-rkdvl...
Getting source from Git repository
32:13
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/bigdata/search/query-processing-module/.git/
Created fresh repository.
fatal: unable to access '<git-repository>': Failed to connect to <gitlab-url> port 443 after 130010 ms: Operation timed out
Cleaning up project directory and file based variables
30:01
ERROR: Job failed: command terminated with exit code 1

CodePudding user response：

Here is how I would debug this Issue:

1. Make sure there are no NetworkPolicies present, that are restricting the egress of the pod.
2. If you have the newest Kubernetes version you can run an ephemeral debug container inside the Pod to examine the networking situation. Docs

kubectl debug -it ephemeral-demo --image=busybox:1.28 --target=ephemeral-demo

2. If not you can try to get a shell inside your container and examine the situation from there or you can try to start a pod on the same node and try to connect from there.

As soon as you have a shell inside some container that doesn't work try to answer the following questions:

• Can you connect to some other Server?
• Can you resolve the hostname?
• Is the IP a private one and overlapping with some internal Kubernetes IPs?
• Can you ping the IP? If yes
  • Can you curl the IP? If no
    • If you open another port on the target machine can you connect to this port? => if yes probably some firewall problem somewhere
• If no (can't ping) => can be either firewall related or IP routing related.

I cannot say for sure what is wrong, but try the steps above and hopefully you get some insight into where the Problem is.