Ingress nginx stripping grpc request metadata headers-CodePudding

Problem Summary

When using ingress nginx controller to route requests to my grpc server, the request metadata headers get stripped out. When I directly hit the service the headers are present.

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
  - host: <domain name>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: <service name>
            port:
              number: 50051
  tls:
  - hosts:
    - <domain name>
    secretName: <secret>

Troubleshooting I've done.

example command I used to request to nginx. When I inspect the server log, the "test.header" metadata is missing

./grpcurl -vv -cacert <cert path> -d '{<data>}' -H 'test.header:header' --proto <file> -import-path . <domain name>:443 <endpoint>

example command to request directly to the service. The "test.header" metadata comes through

./grpcurl -vv --plaintext -d '{<data>}' -H 'test.header:header' --proto <file> -import-path . <service name>:50051 <endpoint>

Inspecting nginx log with --v=5 gives me this. No error message, but the header is not there.

2022/08/26 02:54:34 [debug] 237#237: *20996 http2 table add: "content-type: application/grpc"
2022/08/26 02:54:34 [debug] 237#237: *20996 http2 header: "content-type: application/grpc"
2022/08/26 02:54:34 [debug] 237#237: *20996 http2 table add: "user-agent: grpcurl/v1.8.7 grpc-go/1.48.0"
2022/08/26 02:54:34 [debug] 237#237: *20996 http2 header: "user-agent: grpcurl/v1.8.7 grpc-go/1.48.0"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":method: POST"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":scheme: http"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":path: <path>"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: ":authority: upstream_balancer"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-request-id: e89c7ad55ebd852a06416158be06f799"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-real-ip: <>"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-for: <>"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-host: <domain name>:443"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-port: 443"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-proto: https"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-forwarded-scheme: https"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "x-scheme: https"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "te: trailers"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "content-type: application/grpc"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: "user-agent: grpcurl/v1.8.7 grpc-go/1.48.0"
2022/08/26 02:54:34 [debug] 237#237: *20996 grpc header: 505249202a20485454502f322e300d0a0d0a534d0d0a0d0a00001204000000000000010000000000020000000000047fffffff0000040800000000007fff0000000136010400000001838644b16087aa534d6d83498f54482d9dcc42afdc2bde3d529a6b6c1a4c7ab716cee62158c454de3d529a6b6c1a4c7aaefacc5f5f418cb6b426c28e98a31d03a885b30089f2b585ed6950958d27972bcf91d1c8db658e479b10c0e341702dbd1940e4aebeff0087f2b5851d0b1abf87081765f5c4bbff008bf2b4a7b3c0ec90b22d29ec87081765f5c4bbff008cf2b4a7b3c0ec90b22d39d09f921d03498f51283db22e6a925b2a1d171a699f008cf2b4a7b3c0ec90b2..., len: 383
...

Looking through different documents, there seem to be some options for http

"nginx.org/proxy-pass-headers" annotation
"proxy_pass_request_headers" flag in nginx configuration: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass_request_headers

But I'm struggling to find anything for grpc. The closest I could find was "grpc_set_header" flag in nginx configuration,

http://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_set_header
https://github.com/kubernetes/ingress-nginx/pull/2524 but I'm not sure if I can set it to the values from the request with this flag, and ideally, I want to just forward all headers from my requests.

I'm very surprised and confused there isn't many documents or posts about what I'm experiencing, which makes me think that I'm doing something wrong. Any help would be appreciated.

CodePudding user response：

Try with the in configmap use-forwarded-headers

use-forwarded-headers: "true"

Doc ref : https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-forwarded-headers

If your header container the underscore you can enable that also in configmap of Nginx ingress controller

enable-underscores-in-headers: true

CodePudding user response：

Update --

My issue was the dots in my header keys. https://trac.nginx.org/nginx/ticket/1558

Nginx was ignoring the keys with dots, and when I removed the dots the headers started coming through. I also tried using "ignore_invalid_headers: false" config as suggested in the nginx ticket above, but that didn't work for me. Just had to update the header keys.