Different hosts (Pods in Kubernetes) responds with different certificate for the same hostname-CodePudding

I have a werid problem - when asking for my internal hostname, xxx.home.arpa via e.g openssl s_client -connect xxx.home.arpa:443 one (example) pod

 -  image: docker.io/library/node:8.17.0-slim
    name: node
    args:
      - "86400"
    command:
      - sleep

is getting response with DEFAULT NGINX INGRESS CERTIFICATE.

Other pod in the same namespace for the same command is getting response with my custom certificate.

Question: Why one pod RECEIVES different cert for the same request?

For the purpose of this problem, please assume that cert-manager and certs should be properly configured - they are working in most of the system, it's only few pods that are misbehaving

Configuration: k8s nginx ingress, calico CNI, custom coredns svc which manages DNS responses (might be important?), my own CA authority.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: ca-issuer
    kubernetes.io/ingress.class: nginx
  creationTimestamp: "2022-03-13T06:54:17Z"
  generation: 1
  name: gerrit-ingress
  namespace: gerrit
  resourceVersion: "739842"
  uid: f22034ab-0ed8-4779-b01e-2738e6f63eb7
spec:
  rules:
  - host: gerrit.home.arpa
    http:
      paths:
      - backend:
          service:
            name: gerrit-gerrit-service
            port:
              number: 80
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - gerrit.home.arpa
    secretName: gerrit-tls
status:
  loadBalancer:
    ingress:
    - ip: 192.168.10.2

Most of the configuration (Except DNS) is up here.

CodePudding user response：

As it turns out, my initial guesses were far off - particular container had a set of tools which were both configured to not send servername (Or not support SNI at all, which was the problem), specifically yarn:1.x and openssl:1.0.x.

The problem was with SNI of course, newer openssl or curl do use -servername by default satisfying SNI requirements.

To this I've considered two solutions:

Wildcard DNS for the clients that do not support SNI, which is easier but does not feel secure
TLS termination with reverse proxy allowing me to transparently use client with SNI support, which I haven't yet tried.

I went with wildcard DNS, though I don't feel that this should be done in prod. :)