ErrImagePull: Cannot pull image from registry.gitlab.com private registry from private GKE cluster-CodePudding

I currently have a private GKE Autopilot cluster with cluster firewall rules that allow it to have access to the internet. I am able to pull public images from quay.io and Docker Hub just fine. I am just not able to pull an image from my private GitLab container registry. I have created a secret of type kubernetes.io/dockerconfigjson and I have added to my deployment.yaml file. I am able to pull images from the private registry when I start a local cluster on my machine with the same secret just fine. Aditionally, I am deploying the yaml files from my laptop to the GKE cluster for testing for now but I also have ArgoCD installed on it to handle the deployments. Here is the error that I am seeing:

Failed to pull image "registry.gitlab.com/my-group/my-project:latest ": rpc error: code = Unknown desc = failed to pull and unpack image "registry.gitlab.com/verity/excel-?calculation-api:latest": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry.gitlab.com/v2/my-group/my-repo/blobs/sha256:: 403 Forbidden

Docker config.json

{
    "auths": {
        "registry.gitlab.com": {
            "auth": "base64-encoded-creds-in-username:password-format"
        }
    }
}

gitlab-secret.yaml:

apiVersion: v1
kind: Secret
type: kubernetes.io/dockerconfigjson
metadata:
  name: gitlab-secret
data:
  .dockerconfigjson: <base-64-string>

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-api
spec:
  selector:
    matchLabels:
      app: my-api
  template:
    metadata:
      labels:
        app: my-api
    spec:
      imagePullSecrets:
      - name: gitlab-secret
      containers:
      - name: my-api
        image: registry.gitlab.com/my-group/my-repo
        imagePullPolicy: Always
        resources:
          limits:
            cpu: 1000m
            memory: 6Gi
          requests:
            cpu: 500m
            memory: 3Gi
        ports:
        - containerPort: 5000

Is there anything I am doing that is clearly wrong that I am doing here?

CodePudding user response：

Are you able to login to the registry using the credentials that are used in ImagePullSecret?

docker login registry.gitlab.com
Username: xxxxx
Password: xxxxxx

Then try to pull the image using below command

docker pull registry.gitlab.com/my-group/my-repo

Let me know if the above tests are run and successful

CodePudding user response：

For those that may run into the same issue I did, here is the short of it. I had forgotten that I had deployed the cluster behind a service perimeter. GitLab hosts their container registry in Google Cloud and so when you git the registry.gitlab.com api, it redirects it to the storage.googleapis.com api. This was blocked by my service perimeter and hence I could not pull the images in my private repo. If you have a service perimeter set up around your GKE cluster, you will need to configure an egress rule for the service perimeter that allows your project to have access to the GitLab's project on google cloud. In my case I made an egress rule that allowed my to communicate with project: 805818759045 service: storage.googleapis.com using the gitlab-object-storage-prd@gitlab-production.iam.gserviceaccount.com service account.