Kubelet service is not running. It seems like the kubelet isn't running or healthy-CodePudding

I have configured 1 master 2 workers. after installation successfully kubernetes. It is OK with worker1 joining cluster but I can not join worker2 to the cluster because kubelet service is not running. It seems like the kubelet isn't running or healthy

sudo kubectl get nodes:

NAME STATUS ROLES AGE VERSION
master1 Ready control-plane,master 23m v1.22.2
node1 NotReady 4m13s v1.22.2

I want to know why the kubelet service is not running.

Here kubelet logs.

The start-up result is RESULT.
Dec 04 20:21:26 node2 kubelet[25435]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Dec 04 20:21:26 node2 kubelet[25435]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.659131   25435 server.go:440] "Kubelet version" kubeletVersion="v1.22.2"
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.659587   25435 server.go:868] "Client rotation is on, will bootstrap in background"
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.678863   25435 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.684321   25435 dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.728096   25435 server.go:687] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.728320   25435 container_manager_linux.go:280] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.728388   25435 container_manager_linux.go:285] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName:
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.729329   25435 topology_manager.go:133] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="c
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.729345   25435 container_manager_linux.go:320] "Creating device plugin manager" devicePluginEnabled=true
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.729367   25435 state_mem.go:36] "Initialized new in-memory state store"
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.729408   25435 kubelet.go:314] "Using dockershim is deprecated, please consider using a full-fledged CRI implementation"
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.729430   25435 client.go:78] "Connecting to docker on the dockerEndpoint" endpoint="unix:///var/run/docker.sock"
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.729441   25435 client.go:97] "Start docker client with request timeout" timeout="2m0s"
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.744324   25435 docker_service.go:566] "Hairpin mode is set but kubenet is not enabled, falling back to HairpinVeth" hairpinMode=promiscu
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.744354   25435 docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-veth
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.744554   25435 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.750011   25435 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.750260   25435 docker_service.go:257] "Docker cri networking managed by the network plugin" networkPluginName="cni"
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.753050   25435 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.764080   25435 docker_service.go:264] "Docker Info" dockerInfo=&{ID:4UUR:AFJU:SXYE:5IRP:6G6B:SFDY:H3AA:D5ZB:JSDO:GXVQ:UYNG:POJY Containe
Dec 04 20:21:26 node2 kubelet[25435]: E1204 20:21:26.765777   25435 server.go:294] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" i
Dec 04 20:21:26 node2 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Dec 04 20:21:26 node2 systemd[1]: kubelet.service: Failed with result 'exit-code'.

kubeadm join logs:

I1204 20:27:56.222794   29796 join.go:405] [preflight] found NodeName empty; using OS hostname as NodeName
I1204 20:27:56.223032   29796 initconfiguration.go:116] detected and using CRI socket: /var/run/dockershim.sock
[preflight] Running pre-flight checks
I1204 20:27:56.223834   29796 preflight.go:92] [preflight] Running general checks
I1204 20:27:56.225983   29796 checks.go:245] validating the existence and emptiness of directory /etc/kubernetes/manifests
I1204 20:27:56.226133   29796 checks.go:282] validating the existence of file /etc/kubernetes/kubelet.conf
I1204 20:27:56.226271   29796 checks.go:282] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I1204 20:27:56.226408   29796 checks.go:106] validating the container runtime
I1204 20:27:56.282374   29796 checks.go:132] validating if the "docker" service is enabled and active
I1204 20:27:56.300100   29796 checks.go:331] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I1204 20:27:56.300279   29796 checks.go:331] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1204 20:27:56.300580   29796 checks.go:649] validating whether swap is enabled or not
I1204 20:27:56.300738   29796 checks.go:372] validating the presence of executable conntrack
I1204 20:27:56.301009   29796 checks.go:372] validating the presence of executable ip
I1204 20:27:56.301613   29796 checks.go:372] validating the presence of executable iptables
I1204 20:27:56.301801   29796 checks.go:372] validating the presence of executable mount
I1204 20:27:56.302057   29796 checks.go:372] validating the presence of executable nsenter
I1204 20:27:56.302384   29796 checks.go:372] validating the presence of executable ebtables
I1204 20:27:56.302473   29796 checks.go:372] validating the presence of executable ethtool
I1204 20:27:56.302569   29796 checks.go:372] validating the presence of executable socat
I1204 20:27:56.302610   29796 checks.go:372] validating the presence of executable tc
I1204 20:27:56.303072   29796 checks.go:372] validating the presence of executable touch
I1204 20:27:56.303472   29796 checks.go:520] running all checks
I1204 20:27:56.372402   29796 checks.go:403] checking whether the given node name is valid and reachable using net.LookupHost
I1204 20:27:56.373211   29796 checks.go:618] validating kubelet version
I1204 20:27:56.467792   29796 checks.go:132] validating if the "kubelet" service is enabled and active
I1204 20:27:56.485715   29796 checks.go:205] validating availability of port 10250
I1204 20:27:56.486624   29796 checks.go:282] validating the existence of file /etc/kubernetes/pki/ca.crt
I1204 20:27:56.487016   29796 checks.go:432] validating if the connectivity type is via proxy or direct
I1204 20:27:56.487841   29796 join.go:475] [preflight] Discovering cluster-info
I1204 20:27:56.488260   29796 token.go:80] [discovery] Created cluster-info discovery client, requesting info from "192.168.1.53:6443"
I1204 20:27:56.520182   29796 token.go:118] [discovery] Requesting info from "192.168.1.53:6443" again to validate TLS against the pinned public key
I1204 20:27:56.530589   29796 token.go:135] [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.1.53:6443"
I1204 20:27:56.530702   29796 discovery.go:52] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process
I1204 20:27:56.530924   29796 join.go:489] [preflight] Fetching init configuration
I1204 20:27:56.531171   29796 join.go:534] [preflight] Retrieving KubeConfig objects
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
I1204 20:27:56.549808   29796 interface.go:431] Looking for default routes with IPv4 addresses
I1204 20:27:56.549913   29796 interface.go:436] Default route transits interface "enp0s3"
I1204 20:27:56.550259   29796 interface.go:208] Interface enp0s3 is up
I1204 20:27:56.550564   29796 interface.go:256] Interface "enp0s3" has 2 addresses :[192.168.1.50/24 fe80::a00:27ff:fe7e:db8b/64].
I1204 20:27:56.550644   29796 interface.go:223] Checking addr  192.168.1.50/24.
I1204 20:27:56.550887   29796 interface.go:230] IP found 192.168.1.50
I1204 20:27:56.550955   29796 interface.go:262] Found valid IPv4 address 192.168.1.50 for interface "enp0s3".
I1204 20:27:56.551237   29796 interface.go:442] Found active IP 192.168.1.50
I1204 20:27:56.563573   29796 preflight.go:103] [preflight] Running configuration dependant checks
I1204 20:27:56.563872   29796 controlplaneprepare.go:219] [download-certs] Skipping certs download
I1204 20:27:56.565399   29796 kubelet.go:112] [kubelet-start] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf
I1204 20:27:56.569613   29796 kubelet.go:120] [kubelet-start] writing CA certificate at /etc/kubernetes/pki/ca.crt
I1204 20:27:56.572216   29796 kubelet.go:141] [kubelet-start] Checking for an existing Node in the cluster with name "node2" and status "Ready"
I1204 20:27:56.576685   29796 kubelet.go:155] [kubelet-start] Stopping the kubelet
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
I1204 20:28:01.956734   29796 kubelet.go:190] [kubelet-start] preserving the crisocket information for the node
I1204 20:28:01.956911   29796 patchnode.go:31] [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "node2" as an annotation
I1204 20:28:01.957066   29796 cert_rotation.go:137] Starting client certificate rotation controller
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.

CodePudding user response：

First, check if swap is diabled on your node as you MUST disable swap in order for the kubelet to work properly.

sudo swapoff -a  
sudo sed -i '/ swap / s/^/#/' /etc/fstab

Also check out if kubernetes and docker cgroup driver is set to same. From kubernetes documentation:

Both the container runtime and the kubelet have a property called "cgroup driver", which is important for the management of cgroups on Linux machines.
Warning: Matching the container runtime and kubelet cgroup drivers is required or otherwise the kubelet process will fail.

The Container runtimes page explains that the systemd driver is recommended for kubeadm based setups instead of the cgroupfs driver, because kubeadm manages the kubelet as a systemd service.

For docker:

docker info |grep -i cgroup

You can add this to /etc/docker/daemon.json to set the docker cgroup driver to systemd:

{
    "exec-opts": ["native.cgroupdriver=systemd"]
}

Restart your docker service after making any changes with

sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl restart kubelet

You can try to execute kubeadm join after performing the above steps.