I'm currently trying to setup an application in K8S behind an nginx-ingress. The Certs should be generated by cert-manager and Let's Encrypt (Staging for now).
The application is in namespace prod, nginx-ingress-controller in namespace nginx and cert-manager lives in cert-manager namespace.
We setup a ClusterIssuer for Let's Encrypt staging and successfully generated a certificate (we can see it in the secrets and certificate resource). However, nginx-ingress-controller is still answering with the Kubernetes Ingress Controller Fake Certificate.
Here are some technical details:
Ingress
❯ kubectl describe ingress/forgerock
Name: forgerock
Labels: <none>
Namespace: prod
Address: someaws-id.elb.eu-central-1.amazonaws.com
Ingress Class: <none>
Default backend: <default>
TLS:
sslcertciam terminates ciam.test.fancycorp.com
Rules:
Host Path Backends
---- ---- --------
ciam.test.fancycorp.com
/am/json/authenticate am:80 (10.0.2.210:8081)
...
/am/extlogin am:80 (10.0.2.210:8081)
Annotations: cert-manager.io/cluster-issuer: letsencrypt-stage
haproxy.router.openshift.io/cookie_name: route
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/body-size: 64m
nginx.ingress.kubernetes.io/enable-cors: false
nginx.ingress.kubernetes.io/proxy-body-size: 64m
nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
nginx.ingress.kubernetes.io/proxy-read-timeout: 600
nginx.ingress.kubernetes.io/proxy-send-timeout: 600
nginx.ingress.kubernetes.io/send-timeout: 600
nginx.ingress.kubernetes.io/session-cookie-hash: sha1
nginx.ingress.kubernetes.io/session-cookie-name: route
nginx.ingress.kubernetes.io/ssl-redirect: true
Events: <none>
Issuer:
❯ kubectl describe clusterissuer/letsencrypt-stage
Name: letsencrypt-stage
Namespace:
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1
Kind: ClusterIssuer
Metadata:
Creation Timestamp: 2022-09-12T07:26:05Z
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:acme:
.:
f:email:
f:privateKeySecretRef:
.:
f:name:
f:server:
f:solvers:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-09-12T07:26:05Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:acme:
.:
f:lastRegisteredEmail:
f:uri:
f:conditions:
Manager: controller
Operation: Update
Subresource: status
Time: 2022-09-12T07:26:06Z
Resource Version: 17749318
UID: fcbcbfff-b875-4ac4-805b-65ab0b4e1a93
Spec:
Acme:
Email: [email protected]
Preferred Chain:
Private Key Secret Ref:
Name: letsencrypt-stage
Server: https://acme-staging-v02.api.letsencrypt.org/directory
Solvers:
http01:
Ingress:
Class: nginx
Status:
Acme:
Last Registered Email: [email protected]
Uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/68184363
Conditions:
Last Transition Time: 2022-09-12T07:26:06Z
Message: The ACME account was registered with the ACME server
Observed Generation: 1
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events: <none>
Certificate:
❯ kubectl describe cert/sslcertciam
Name: sslcertciam
Namespace: prod
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2022-09-12T07:40:04Z
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:ownerReferences:
.:
k:{"uid":"2a0af8f2-8166-4a8e-bb50-fd0aa906f844"}:
f:spec:
.:
f:dnsNames:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:secretName:
f:usages:
Manager: controller
Operation: Update
Time: 2022-09-12T07:40:04Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
f:notAfter:
f:notBefore:
f:renewalTime:
f:revision:
Manager: controller
Operation: Update
Subresource: status
Time: 2022-09-12T07:40:07Z
Owner References:
API Version: networking.k8s.io/v1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: forgerock
UID: 2a0af8f2-8166-4a8e-bb50-fd0aa906f844
Resource Version: 17753197
UID: 2484d1fe-5b80-4cbc-b2f8-7f4276e15a37
Spec:
Dns Names:
ciam.test.fancycorp.com
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-stage
Secret Name: sslcertciam
Usages:
digital signature
key encipherment
Status:
Conditions:
Last Transition Time: 2022-09-12T07:40:07Z
Message: Certificate is up to date and has not expired
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Not After: 2022-12-11T06:40:05Z
Not Before: 2022-09-12T06:40:06Z
Renewal Time: 2022-11-11T06:40:05Z
Revision: 1
Events: <none>
Secret:
❯ kubectl describe secret/sslcertciam
Name: sslcertciam
Namespace: prod
Labels: <none>
Annotations: cert-manager.io/alt-names: ciam.test.fancycorp.com
cert-manager.io/certificate-name: sslcertciam
cert-manager.io/common-name: ciam.test.fancycorp.com
cert-manager.io/ip-sans:
cert-manager.io/issuer-group: cert-manager.io
cert-manager.io/issuer-kind: ClusterIssuer
cert-manager.io/issuer-name: letsencrypt-stage
cert-manager.io/uri-sans:
Type: kubernetes.io/tls
Data
====
tls.crt: 5741 bytes
tls.key: 1675 bytes
Certificate Request:
❯ kubectl describe certificaterequests/sslcertciam-p6qpg
Name: sslcertciam-p6qpg
Namespace: prod
Labels: <none>
Annotations: cert-manager.io/certificate-name: sslcertciam
cert-manager.io/certificate-revision: 1
cert-manager.io/private-key-secret-name: sslcertciam-ztc8q
API Version: cert-manager.io/v1
Kind: CertificateRequest
Metadata:
Creation Timestamp: 2022-09-12T07:40:05Z
Generate Name: sslcertciam-
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:cert-manager.io/certificate-name:
f:cert-manager.io/certificate-revision:
f:cert-manager.io/private-key-secret-name:
f:generateName:
f:ownerReferences:
.:
k:{"uid":"2484d1fe-5b80-4cbc-b2f8-7f4276e15a37"}:
f:spec:
.:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:request:
f:usages:
Manager: controller
Operation: Update
Time: 2022-09-12T07:40:05Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:certificate:
f:conditions:
Manager: controller
Operation: Update
Subresource: status
Time: 2022-09-12T07:40:06Z
Owner References:
API Version: cert-manager.io/v1
Block Owner Deletion: true
Controller: true
Kind: Certificate
Name: sslcertciam
UID: 2484d1fe-5b80-4cbc-b2f8-7f4276e15a37
Resource Version: 17753174
UID: 2289de7b-f43f-4859-816b-b4a9794846ec
Spec:
Extra:
authentication.kubernetes.io/pod-name:
cert-manager-75947cd847-7gndz
authentication.kubernetes.io/pod-uid:
91415540-9113-4456-86d2-a0e28478718a
Groups:
system:serviceaccounts
system:serviceaccounts:cert-manager
system:authenticated
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-stage
Request: xxx
UID: 5be755b9-711c-49ac-a962-6b3a3f80d16e
Usages:
digital signature
key encipherment
Username: system:serviceaccount:cert-manager:cert-manager
Status:
Certificate: <base64-encoded-cert>
Conditions:
Last Transition Time: 2022-09-12T07:40:05Z
Message: Certificate request has been approved by cert-manager.io
Reason: cert-manager.io
Status: True
Type: Approved
Last Transition Time: 2022-09-12T07:40:06Z
Message: Certificate fetched from issuer successfully
Reason: Issued
Status: True
Type: Ready
Events: <none>
Curl:
❯ curl -v https://ciam.test.fancycorp.com/am/extlogin/ -k
* Trying xxx.xxx.xxx.xxx:443...
* Connected to ciam.test.fancycorp.com (xxx.xxx.xxx.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* start date: Sep 12 07:43:15 2022 GMT
* expire date: Sep 12 07:43:15 2023 GMT
* issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x126811e00)
> GET /am/extlogin/ HTTP/2
> Host: ciam.test.fancycorp.com
> user-agent: curl/7.79.1
> accept: */*
...
Update 1:
When running kubectl ingress-nginx certs --host ciam.test.fancycorp.com, I am also getting the Fake Certificate returned.
CodePudding user response：
Found the issue and solution...
There was another ingress defined in another namespace that did define the same hostname, but failed to link to a proper secret with the TLS cert. When I deleted that one, it immediately worked.
Lessons learned: Be aware of impacts from other namespaces!