Unable to launch new pods despite resources seemingly available-CodePudding

As the title indicates I'm unable to launch new pods despite ressources seemingly being available.

Judging from the below screenshot there should be room for about 40 new pods.

And also judging from the following screenshot the nodes seems fairly underutilised

However I'm currently facing the below error message

0/3 nodes are available: 1 Insufficient cpu, 2 node(s) had volume node affinity conflict.

And last night it was the following

0/3 nodes are available: 1 Too many pods, 2 node(s) had volume node affinity conflict.

Most of my services require very little memory and cpu. And therefore their resources are configured as seen below

resources:
  limits:
    cpu: 100m
    memory: 64Mi
  requests:
    cpu: 100m
    memory: 32Mi

Can someone help me understand why I can't deploy more pods? And how to fix this?

(Also, please let me know if I should post additional information)

CodePudding user response：

volume node affinity conflict - the volume you tried to mount is not available on any of the node. You can resolve this or paste your volumes section to the question for further examination.

CodePudding user response：

Can someone help me understand why I can't deploy more pods?

Your problem is volume node affinity conflict.

The error "volume node affinity conflict" happens when the persistent volume claims that the pod is using are scheduled on different zones, rather than on one zone, and so the actual pod was not able to be scheduled because it cannot connect to the volume from another zone.

And how to fix this?

First, try to investigate exactly where the problem is. You can find a detailed guide here. You will need commands like:

kubectl get pv  
kubectl describe pv  
kubectl get pvc  
kubectl describe pvc

Then you can delete the PV and PVC and move pods to the same zone along with the PV and PVC.

Look also at this similar problem. There you find a couple of solutions how can you solve your issue.