Kubelet pulls image even though image digest is already present on Node-CodePudding

I have a small script running in a Pod that pokes for the latest App images (dashboard:development) in my registry and then pushes them to the Nodes running (via a daemonset).

This does work, as seen below.

Now, I would assume that once an App pod (like sp-pod-xx) requests this image, kubelet should not try to re-pull the image, even if imagePullPolicy: Always is set. As the docs say, kubelet compares the digest and only pulls, if there is a mismatch:

Always: every time the kubelet launches a container, the kubelet queries the container image registry to resolve the name to an image digest. If the kubelet has a container image with that exact digest cached locally, the kubelet uses its cached image; otherwise, the kubelet pulls the image with the resolved digest, and uses that image to launch the container.

But, even though the digests are identical (I did verify this), kubelet still re-pulls the image anyway. The App pod and the Daemonset pods are running on the same nodes too.

Any idea why?

Event logs:

4m5s        Normal   Killing             pod/image-puller-ds-ldbfz                         
3m57s       Normal   SuccessfulCreate    daemonset/image-puller-ds                         Created pod: image-puller-ds-fcmts
3m57s       Normal   SuccessfulCreate    daemonset/image-puller-ds                         Created pod: image-puller-ds-fhhds
3m57s       Normal   Pulled              pod/image-puller-ds-fhhds                         Successfully pulled image "dashboard:development" in 192.717161ms
3m57s       Normal   Pulling             pod/image-puller-ds-fhhds                         Pulling image "dashboard:development"
3m56s       Normal   Started             pod/image-puller-ds-fhhds                         Started container image-puller
3m56s       Normal   Created             pod/image-puller-ds-fcmts                         Created container image-puller
3m56s       Normal   Created             pod/image-puller-ds-fhhds                         Created container image-puller
3m56s       Normal   Started             pod/image-puller-ds-fcmts                         Started container image-puller
3m56s       Normal   Pulled              pod/image-puller-ds-fhhds                         Container image "pause:0.0.1" already present on machine
3m55s       Normal   Created             pod/image-puller-ds-fcmts                         Created container pause
3m55s       Normal   SuccessfulDelete    daemonset/image-puller-ds                         Deleted pod: image-puller-ds-xt9vv
3m55s       Normal   Pulled              pod/image-puller-ds-fcmts                         Container image "pause:0.0.1" already present on machine
3m55s       Normal   Created             pod/image-puller-ds-fhhds                         Created container pause
3m55s       Normal   Started             pod/image-puller-ds-fhhds                         Started container pause
3m55s       Normal   Started             pod/image-puller-ds-fcmts                         Started container pause
3m55s       Normal   Killing             pod/image-puller-ds-xt9vv                         Stopping container pause
3m54s       Normal   Killing             pod/image-puller-ds-wgwzh                         Stopping container pause
3m54s       Normal   SuccessfulDelete    daemonset/image-puller-ds                         Deleted pod: image-puller-ds-wgwzh
3m25s       Normal   Pulling             pod/sp-pod-f3884032-1164-48e8-8213-c0c3856e573d   Pulling image "dashboard:development"
3m25s       Normal   Pulled              pod/sp-pod-f3884032-1164-48e8-8213-c0c3856e573d   Successfully pulled image "dashboard:development" in 220.610781ms
3m25s       Normal   Created             pod/sp-pod-f3884032-1164-48e8-8213-c0c3856e573d   Created container sp-container-f3884032-1164-48e8-8213-c0c3856e573d
3m25s       Normal   Started             pod/sp-pod-f3884032-1164-48e8-8213-c0c3856e573d   Started container sp-container-f3884032-1164-48e8-8213-c0c3856e573d

Versions:

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"2021-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.12", GitCommit:"f941a31f4515c5ac03f5fc7ccf9a330e3510b80d", GitTreeState:"clean", BuildDate:"2022-11-09T17:12:33Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

CodePudding user response：

Use imagePullPolicy: IfNotPresent in the pod spec. to utilize the image that is already present on the node.

Use imagePullPolicy: Always to pull fresh image from image registry.

CodePudding user response：

The docs are wrong. You can read the source code here. The policy PullIfNotPresent would check if the image exists, but Always skips directly to pulling the image.