Docker DinD fails to build images after upgrading-CodePudding

We are using Docker 18.9.8-dind. DinD — Docker-in-Docker — is running Docker in a separate container. This way, we send requests to this container to build our images, instead of executing Docker in the machine that wants the built image.

We needed to upgrade from 18.9.8-dind to 20.10.14-dind. Since we use Kubernetes, we just updated the image version in some YAML files:

     spec:
       containers:
       - name: builder
-        image: docker:18.09.8-dind
         image: docker:20.10.14-dind
         args: ["--storage-driver", "overlay2", "--mtu", "1460"]
         imagePullPolicy: Always
         resources:

Alas, things stopped working after that. Builds failed, and we could find these error messages in the code reaching for our Docker builder:

{"errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"123.456.789.10","port":2375}
Something went wrong and the entire build was interrupted due to an incorrect configuration file or build step,
check your source code.

What can be going on?

CodePudding user response：

We checked the logs in the Docker pod, and found this message at the end:

API listen on [::]:2376

Well, our message in the question states we tried to connect to port 2375, which used to work. Why has the port changed?

Docker enables TLS as default from version 19.03 onwards. When Docker uses TLS, it listens on port 2376.

We had two alternatives here:

change the port to 2375 (which sounds like a bad idea: we would use the default plain port for TLS communication. Which could confuse in the future);
Connect to the new port; or
disable TLs.

For many reasons, we choose to disable TLS, which only requires an environment variable in yet another YAML file:

       - name: builder
         image: docker:20.10.14-dind
         args: ["--storage-driver", "overlay2", "--mtu", "1460"]
         env:
         - name: DOCKER_TLS_CERTDIR
           value: ""
         imagePullPolicy: Always
         resources:
           requests:

In most scenarios, though, it is probably. better to have TLS enabled. Then, change the port on whoever is connecting to Docker.

(Sharing in the spirit of Can I answer my own questions? because it took us some time to piece the parts together. Maybe by sharing this information together with the error message, things can be easier for other affected people to find.)