I am using google container registry (GCR) to push and pull docker images. I have created a deployment in kubernetes with 3 replicas. The deployment will use a docker image pulled from the GCR.

Out of 3 replicas, 2 are pulling the images and running fine.But the third replica is showing the below error and the pod's status remains "ImagePullBackOff" or "ErrImagePull"

"Failed to pull image "gcr.io/xxx:yyy": rpc error: code = Unknown desc = failed to pull and unpack image "gcr.io/xxx:yyy": failed to resolve reference "gcr.io/xxx:yyy": unexpected status code: 401 Unauthorized"

I am confused like why only one of the replicas is showing the error and the other 2 are running without any issue. Can anyone please clarify this?

Thanks in Advance!

CodePudding user response：

Hi you will setup role for cluster to access GCR images for pulling and pushing you can see https://github.com/GoogleContainerTools/skaffold/issues/336

CodePudding user response：

ImagePullBackOff and ErrImagePull indicate that the image used by a container cannot be loaded from the image registry.

401 unauthorized error might occur when you pull an image from a private Container Registry repository. For troubleshooting the error:

1. Identify the node that runs the pod by kubectl describe pod POD_NAME | grep "Node:"

2. Verify the node has the storage scope by running the command

   gcloud compute instances describe NODE_NAME  --zone=COMPUTE_ZONE --format="flattened(serviceAccounts[].scopes)"
   

3. The node's access scope should contain at least one of the following:

   serviceAccounts[0].scopes[0]: https://www.googleapis.com/auth/devstorage.read_only serviceAccounts[0].scopes[0]: https://www.googleapis.com/auth/cloud-platform

4. Recreate the node pool that node belongs to with sufficient scope and you cannot modify existing nodes, you must recreate the node with the correct scope.

   • Create a new node pool with the gke-default scope by the following command

     gcloud container node-pools create NODE_POOL_NAME  --cluster=CLUSTER_NAME  --zone=COMPUTE_ZONE   --scopes="gke-default"
     

   • Create a new node pool with only storage scope

     gcloud container node-pools create NODE_POOL_NAME   --cluster=CLUSTER_NAME  --zone=COMPUTE_ZONE --scopes="https://www.googleapis.com/auth/devstorage.read_only"
     

Refer to the link for more information on the troubleshooting process.