I've been trying to create an EKS cluster with vpc-cni addon due to the pod restrictions for m5.xlarge VMs (57). After creation I can see it is passed to the launchtemplate object but when doing a node describe it still can allocate the previous (wrong?) number
ClusterConfig:
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: exchange-develop
region: us-east-1
version: '1.21'
managedNodeGroups:
- name: default
labels:
worker: default
instanceType: m5.xlarge
desiredCapacity: 2
minSize: 2
maxSize: 4
tags:
'k8s.io/cluster-autoscaler/enabled': 'true'
'k8s.io/cluster-autoscaler/exchange-develop': 'owned'
iam:
attachPolicyARNs:
- arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
- arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
- arn:aws:iam::658464581062:policy/eks-csi-driver-policy
- arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
- arn:aws:iam::658464581062:policy/ALBIngressControllerIAMPolicy
- arn:aws:iam::658464581062:policy/ExternalDNSPlicy
- arn:aws:iam::658464581062:policy/eks-cluster-autoscaler
maxPodsPerNode: 110
availabilityZones: ['us-east-1c', 'us-east-1d']
iam:
withOIDC: true
vpc:
cidr: 10.10.0.0/16
#autoAllocateIPv6: true
# disable public access to endpoint and only allow private access
clusterEndpoints:
publicAccess: true
privateAccess: true
addons:
- name: vpc-cni
version: '1.10.1'
Launch template with redacted data:
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=***
--
Content-Type: text/x-shellscript
Content-Type: charset="us-ascii"
#!/bin/sh
set -ex
sed -i -E "s/^USE_MAX_PODS=\"\\$\{USE_MAX_PODS:-true}\"/USE_MAX_PODS=false/" /etc/eks/bootstrap.sh
KUBELET_CONFIG=/etc/kubernetes/kubelet/kubelet-config.json
echo "$(jq ".maxPods=110" $KUBELET_CONFIG)" > $KUBELET_CONFIG
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex
B64_CLUSTER_CA=<>
API_SERVER_URL=<>
K8S_CLUSTER_DNS_IP=<>
/etc/eks/bootstrap.sh exchange-develop --kubelet-extra-args '--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,alpha.eksctl.io/cluster-name=exchange-develop,alpha.eksctl.io/nodegroup-name=default,eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=default,worker=default,eks.amazonaws.com/sourceLaunchTemplateId=lt-0037c1eab7037898d --max-pods=58' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP --use-max-pods false
Node description:
Name: ip-10-10-19-34.ec2.internal
Roles: <none>
Labels: alpha.eksctl.io/cluster-name=exchange-develop
alpha.eksctl.io/nodegroup-name=default
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=m5.xlarge
beta.kubernetes.io/os=linux
eks.amazonaws.com/capacityType=ON_DEMAND
eks.amazonaws.com/nodegroup=default
eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd
eks.amazonaws.com/sourceLaunchTemplateId=lt-0037c1eab7037898d
eks.amazonaws.com/sourceLaunchTemplateVersion=1
failure-domain.beta.kubernetes.io/region=us-east-1
failure-domain.beta.kubernetes.io/zone=us-east-1c
kubernetes.io/arch=amd64
kubernetes.io/hostname=<<
kubernetes.io/os=linux
node.kubernetes.io/instance-type=m5.xlarge
topology.kubernetes.io/region=us-east-1
topology.kubernetes.io/zone=us-east-1c
worker=default
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Thu, 02 Dec 2021 10:22:20 -0300
Taints: <none>
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:18 -0300 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:18 -0300 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:18 -0300 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:40 -0300 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.10.19.34
ExternalIP: <<
Hostname: <<
InternalDNS: <<
ExternalDNS: <<
Capacity:
attachable-volumes-aws-ebs: 25
cpu: 4
ephemeral-storage: 83873772Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 15921236Ki
pods: 58
Allocatable:
attachable-volumes-aws-ebs: 25
cpu: 3920m
ephemeral-storage: 76224326324
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 14904404Ki
pods: 58
System Info:
Machine ID: ec28ac2717ec395cdf5b4e37f7672569
System UUID: ec28ac27-17ec-395c-df5b-4e37f7672569
Boot ID: 50b3d3d9-5dfa-40b6-99c8-20873632c7ca
Kernel Version: 5.4.156-83.273.amzn2.x86_64
OS Image: Amazon Linux 2
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://20.10.7
Kubelet Version: v1.21.5-eks-bc4871b
Kube-Proxy Version: v1.21.5-eks-bc4871b
ProviderID: aws:///<<<
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system aws-node-9z7pw 25m (0%) 0 (0%) 0 (0%) 0 (0%) 61m
kube-system kube-proxy-2slc8 100m (2%) 0 (0%) 0 (0%) 0 (0%) 61m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 125m (3%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events: <none>
See allocatable pods 58...
So, what is the correct way of using eksctl to create a cluster with vpc-cni and pass the maxPodsPerNode argument to ec2 launch template?
EDIT:
Other things I've tried:
Create cluster from scratch with vpc-cni addon 2 managedNodeGroups with maxPodsPerNode and without, both will take the value of 58 and not 110
Add another nodegroup with eksctl create nodegroup, still 58
Add another nodegroup with EKS AWS UI, still 58
Eksctl version 0.75.0 Kubectl version 1.21.2
CodePudding user response：
For managedNodeGroup you need to specify the AMI ID:
aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.21/amazon-linux-2/recommended/image_id --region us-east-1 --query "Parameter.Value" --output text
managedNodeGroups:
- name: default
...
maxPodsPerNode: 110
ami: ami-00836a7940260f6dd
overrideBootstrapCommand: |
#!/bin/bash
/etc/eks/bootstrap.sh exchange-develop --kubelet-extra-args '--node-labels=eks.amazonaws.com/nodegroup=default,eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd'