Cannot get elastic working via kubernetes ingress-CodePudding

I have the following service:

# kubectl get svc es-kib-opendistro-es-client-service -n logging
NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                               AGE
es-kib-opendistro-es-client-service   ClusterIP   10.233.19.199   <none>        9200/TCP,9300/TCP,9600/TCP,9650/TCP   279d
#

When I perform a curl to the IP address of the service it works fine:

# curl https://10.233.19.199:9200/_cat/health -k --user username:password 
1638224389 22:19:49 elasticsearch green 6 3 247 123 0 0 0 0 - 100.0%
#

I created an ingress so I can access the service from outside:

# kubectl get ingress ingress-elasticsearch -n logging
NAME                    HOSTS                     ADDRESS                               PORTS     AGE
ingress-elasticsearch   elasticsearch.host.com   10.32.200.4,10.32.200.7,10.32.200.8   80, 443   11h
#

When performing a curl to either 10.32.200.4, 10.32.200.7 or 10.32.200.8 I am getting a openresty 502 Bad Gateway response:

$ curl https://10.32.200.7 -H "Host: elasticsearch.host.com" -k
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>openresty/1.15.8.2</center>
</body>
</html>
$

When tailing the pod logs, I am seeing the following when performing the curl command:

# kubectl logs deploy/es-kib-opendistro-es-client -n logging -f
[2021-11-29T22:22:47,026][ERROR][c.a.o.s.s.h.n.OpenDistroSecuritySSLNettyHttpServerTransport] [es-kib-opendistro-es-client-6c8bc96f47-24k2l] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 414554202a20485454502f312e310d0a486f73743a20656c61737469637365617263682e6f6e696f722e636f6d0d0a582d526571756573742d49443a2034386566326661626561323364663466383130323231386639366538643931310d0a582d5265212c2d49503a2031302e33322e3230302e330d0a582d466f727761726465642d466f723a2031302e33322e3230302e330d0a582d466f727761726465642d486f73743a20656c61737469637365617263682e6f6e696f722e636f6d0d0a582d466f727761721235642d506f72743a203434330d0a582d466f727761726465642d50726f746f3a2068747470730d0a582d536368656d653a2068747470730d0a557365722d4167656e743a206375726c2f372e32392e300d0a4163636570743a202a2f2a0d1b0d0a
#

My ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  labels:
    app: elasticsearch
  name: ingress-elasticsearch
  namespace: logging
spec:
  rules:
  - host: elasticsearch.host.com
    http:
      paths:
      - backend:
          serviceName: es-kib-opendistro-es-client-service
          servicePort: 9200
        path: /
  tls:
  - hosts:
    - elasticsearch.host.com
    secretName: cred-secret
status:
  loadBalancer:
    ingress:
    - ip: 10.32.200.4
    - ip: 10.32.200.7
    - ip: 10.32.200.8

My service:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: es-kib-opendistro-es
    chart: opendistro-es-1.9.0
    heritage: Tiller
    release: es-kib
    role: client
  name: es-kib-opendistro-es-client-service
  namespace: logging
spec:
  clusterIP: 10.233.19.199
  ports:
  - name: http
    port: 9200
    protocol: TCP
    targetPort: 9200
  - name: transport
    port: 9300
    protocol: TCP
    targetPort: 9300
  - name: metrics
    port: 9600
    protocol: TCP
    targetPort: 9600
  - name: rca
    port: 9650
    protocol: TCP
    targetPort: 9650
  selector:
    role: client
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

What is wrong with my setup?

CodePudding user response：

By default, the ingress controller proxies incoming requests to your backend using the HTTP protocol.

You backend service is expecting requests in HTTPS though, so you need to tell nginx ingress controller to use HTTPS.

You can do so by adding an annotation to the Ingress resource like this:

nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

Details about this annotation are in the documentation:

Using backend-protocol annotations is possible to indicate how NGINX should communicate with the backend service. (Replaces secure-backends in older versions) Valid Values: HTTP, HTTPS, GRPC, GRPCS, AJP and FCGI

By default NGINX uses HTTP.