TLS certification expired only for some users-CodePudding

I have a k8s cluster with an ingress nginx as a reverse proxy. I am using letsencrypt to generate TLS certificate

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ******
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
        ingress:
          class: nginx

Everything worked fine for months. Today,

$ curl -v --verbose https://myurl

returns

* Rebuilt URL to: https://myurl/
*   Trying 51.103.58.**...
* TCP_NODELAY set
* Connected to myurl (51.103.58.**) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.

For 2 other people on my team, error is the same and I have the same error when I use Postman (expired certificate).

But for another one, we get no error :

*   Trying 51.103.58.**...
* TCP_NODELAY set
* Connected to myurl (51.103.58.**) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=myurl
*  start date: Jul 24 07:15:13 2021 GMT
*  expire date: Oct 22 07:15:11 2021 GMT
*  subjectAltName: host "myurl" matched cert's "myurl"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd9be00d600)
> GET / HTTP/2
> Host: myurl
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< server: nginx/1.19.1
< date: Thu, 30 Sep 2021 16:11:23 GMT
< content-type: application/json; charset=utf-8
< content-length: 56
< vary: Origin, Accept-Encoding
< access-control-allow-credentials: true
< x-xss-protection: 1; mode=block
< x-frame-options: DENY
< strict-transport-security: max-age=15724800; includeSubDomains
< x-download-options: noopen
< x-content-type-options: nosniff
< etag: W/"38-3eQD3G7Y0vTkrLR ExD2u5BSsMc"
<
* Connection #0 to host myurl left intact
{"started":"2021-09-30T13:30:30.912Z","uptime":9653.048}* Closing connection 0

When I use my web browser to go to the website, everything works fine and the certificate is presented as valid and for now, I get no error in prod or staging environment. (same error on staging)

Has anyone an explanation on this ?

CodePudding user response：

Warning! Please plan OS upgrade path. The below advice should be applied only in emergency situation to quickly fix a critical system.

Your team missed OS update or ca-certificates package update. Below solution works on old Debian/Ubuntu systems.

First check if you have offending DST Root CA X3 cert present:

# grep X3 /etc/ca-certificates.conf 
mozilla/DST_Root_CA_X3.crt

Make sure the client OS have the proper ISRG Root X1 present too:

# grep X1 /etc/ca-certificates.conf 
mozilla/ISRG_Root_X1.crt

This is going to disable X3:

# sed -i '/^mozilla\/DST_Root_CA_X3/s/^/!/' /etc/ca-certificates.conf && update-ca-certificates -f

Try curl https://yourdomain now, should pass.

Again, plan an upgrade please.

CodePudding user response：

Don't have enouigh rep. for just a comment, so I re-post here my answer on Sudden OpenSSL Error messages: error:14090086 using file_get_contents

I fixed the same issue by upgarding my certbot and reissuing certificate with `` --preferred-chain 'ISRG Root X1'

you can do the same with options in the yaml of the cert issuer : see here : https://cert-manager.io/docs/configuration/acme/#use-an-alternative-certificate-chain

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    preferredChain: "ISRG Root X1"