Kubernetes Nginx-Ingress Host-Based Routing works, while Path Based Routing doesn't work-CodePudding

I am running a 2-node K8s cluster on OVH Bare Metal Servers. I've set up MetalLB and Nginx-Ingress.The 2 servers both have public IPs and are not in the same network segment. I've used one of the IPs as the entrypoint for the LB. The deployments I created 3 nginx containers & services to test the forwarding. When I use host based routing, the endpoints are reachable via the internet, but when I use path based forwarding, only the / path is reachable. For the rest, I get the default backend. My host based Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-resource-2
spec:
  ingressClassName: nginx
  rules:
  - host: nginx.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-deploy-main
            port:
              number: 80
  - host: blue.nginx.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-deploy-blue
            port:
              number: 80
  - host: green.nginx.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-deploy-green
            port:
              number: 80

The path based Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-resource-3
spec:
  ingressClassName: nginx
  rules:
  - host: nginx.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80
      - path: /blue
        pathType: Prefix
        backend:
          service:
            name: nginx-deploy-blue
            port:
              number: 80
      - path: /green
        pathType: Prefix
        backend:
          service:
            name: nginx-deploy-green
            port:
              number: 80

The endpoints are all reachable in both cases

# kubectl describe ing ingress-resource-2
Name:             ingress-resource-2
Labels:           <none>
Namespace:        default
Address:          
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host                     Path  Backends
  ----                     ----  --------
  nginx.example.com        
                           /   nginx:80 (192.168.107.4:80)
  blue.nginx.example.com   
                           /   nginx-deploy-blue:80 (192.168.164.212:80)
  green.nginx.example.com  
                           /   nginx-deploy-green:80 (192.168.164.213:80)
Annotations:               <none>
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  13m   nginx-ingress-controller  Configuration for default/ingress-resource-2 was added or updated

# kubectl describe ing ingress-resource-3
Name:             ingress-resource-3
Labels:           <none>
Namespace:        default
Address:          
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host               Path  Backends
  ----               ----  --------
  nginx.example.com  
                     /        nginx:80 (192.168.107.4:80)
                     /blue    nginx-deploy-blue:80 (192.168.164.212:80)
                     /green   nginx-deploy-green:80 (192.168.164.213:80)
Annotations:         <none>
Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  109s  nginx-ingress-controller  Configuration for default/ingress-resource-3 was added or updated

Getting the Nginx-Ingress logs:

# kubectl -n nginx-ingress logs pod/nginx-ingress-6947fb84d4-m9gkk

W0803 17:00:48.516628       1 flags.go:273] Ignoring unhandled arguments: []
I0803 17:00:48.516688       1 flags.go:190] Starting NGINX Ingress Controller Version=2.3.0 PlusFlag=false
I0803 17:00:48.516692       1 flags.go:191] Commit=979db22d8065b22fedb410c9b9c5875cf0a6dc66 Date=2022-07-12T08:51:24Z DirtyState=false Arch=linux/amd64 Go=go1.18.3
I0803 17:00:48.527699       1 main.go:210] Kubernetes version: 1.24.3
I0803 17:00:48.531079       1 main.go:326] Using nginx version: nginx/1.23.0
2022/08/03 17:00:48 [notice] 26#26: using the "epoll" event method
2022/08/03 17:00:48 [notice] 26#26: nginx/1.23.0
2022/08/03 17:00:48 [notice] 26#26: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
2022/08/03 17:00:48 [notice] 26#26: OS: Linux 5.15.0-41-generic
2022/08/03 17:00:48 [notice] 26#26: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2022/08/03 17:00:48 [notice] 26#26: start worker processes
2022/08/03 17:00:48 [notice] 26#26: start worker process 27
2022/08/03 17:00:48 [notice] 26#26: start worker process 28
2022/08/03 17:00:48 [notice] 26#26: start worker process 29
2022/08/03 17:00:48 [notice] 26#26: start worker process 30
2022/08/03 17:00:48 [notice] 26#26: start worker process 31
2022/08/03 17:00:48 [notice] 26#26: start worker process 32
2022/08/03 17:00:48 [notice] 26#26: start worker process 33
2022/08/03 17:00:48 [notice] 26#26: start worker process 34
I0803 17:00:48.543403       1 listener.go:54] Starting Prometheus listener on: :9113/metrics
2022/08/03 17:00:48 [notice] 26#26: start worker process 35
2022/08/03 17:00:48 [notice] 26#26: start worker process 37
I0803 17:00:48.543712       1 leaderelection.go:248] attempting to acquire leader lease nginx-ingress/nginx-ingress-leader-election...
2022/08/03 17:00:48 [notice] 26#26: start worker process 38
...
2022/08/03 17:00:48 [notice] 26#26: start worker process 86
I0803 17:00:48.645253       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"delivery-ingress", UID:"23f93b2d-c3c8-48eb-a2a1-e2ce0453677f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1527358", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/delivery-ingress was added or updated 
I0803 17:00:48.645512       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated 
I0803 17:00:48.646550       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated 
I0803 17:00:48.646629       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"delivery-ingress", UID:"23f93b2d-c3c8-48eb-a2a1-e2ce0453677f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1527358", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/delivery-ingress was added or updated 
I0803 17:00:48.646810       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated 
I0803 17:00:48.646969       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated 
I0803 17:00:48.647259       1 event.go:285] Event(v1.ObjectReference{Kind:"Secret", Namespace:"nginx-ingress", Name:"default-server-secret", UID:"d8271053-2785-408f-b87b-88b9bb9fc488", APIVersion:"v1", ResourceVersion:"1612716", FieldPath:""}): type: 'Normal' reason: 'Updated' the special Secret nginx-ingress/default-server-secret was updated
2022/08/03 17:00:48 [notice] 26#26: signal 1 (SIGHUP) received from 88, reconfiguring
2022/08/03 17:00:48 [notice] 26#26: reconfiguring
2022/08/03 17:00:48 [notice] 26#26: using the "epoll" event method
2022/08/03 17:00:48 [notice] 26#26: start worker processes
2022/08/03 17:00:48 [notice] 26#26: start worker process 89
2022/08/03 17:00:48 [notice] 26#26: start worker process 90
...
2022/08/03 17:00:48 [notice] 26#26: start worker process 136
2022/08/03 17:00:48 [notice] 27#27: gracefully shutting down
2022/08/03 17:00:48 [notice] 27#27: exiting
2022/08/03 17:00:48 [notice] 35#35: gracefully shutting down
2022/08/03 17:00:48 [notice] 31#31: exiting
2022/08/03 17:00:48 [notice] 38#38: gracefully shutting down
2022/08/03 17:00:48 [notice] 32#32: exiting
2022/08/03 17:00:48 [notice] 30#30: exiting
2022/08/03 17:00:48 [notice] 40#40: gracefully shutting down
2022/08/03 17:00:48 [notice] 35#35: exiting
2022/08/03 17:00:48 [notice] 45#45: gracefully shutting down
2022/08/03 17:00:48 [notice] 40#40: exiting
2022/08/03 17:00:48 [notice] 48#48: gracefully shutting down
2022/08/03 17:00:48 [notice] 47#47: exiting
2022/08/03 17:00:48 [notice] 57#57: gracefully shutting down
2022/08/03 17:00:48 [notice] 52#52: exiting
2022/08/03 17:00:48 [notice] 55#55: gracefully shutting down
2022/08/03 17:00:48 [notice] 55#55: exiting
2022/08/03 17:00:48 [notice] 51#51: gracefully shutting down
2022/08/03 17:00:48 [notice] 51#51: exiting
2022/08/03 17:00:48 [notice] 31#31: exit
2022/08/03 17:00:48 [notice] 34#34: gracefully shutting down
2022/08/03 17:00:48 [notice] 34#34: exiting
2022/08/03 17:00:48 [notice] 41#41: exiting
2022/08/03 17:00:48 [notice] 49#49: gracefully shutting down
....
2022/08/03 17:00:48 [notice] 49#49: exiting
2022/08/03 17:00:48 [notice] 57#57: exit
.....
2022/08/03 17:00:48 [notice] 43#43: exit
2022/08/03 17:00:48 [notice] 58#58: gracefully shutting down
2022/08/03 17:00:48 [notice] 38#38: exiting
2022/08/03 17:00:48 [notice] 53#53: gracefully shutting down
2022/08/03 17:00:48 [notice] 48#48: exiting
2022/08/03 17:00:48 [notice] 59#59: gracefully shutting down
2022/08/03 17:00:48 [notice] 58#58: exiting
2022/08/03 17:00:48 [notice] 62#62: gracefully shutting down
2022/08/03 17:00:48 [notice] 60#60: gracefully shutting down
2022/08/03 17:00:48 [notice] 53#53: exiting
2022/08/03 17:00:48 [notice] 61#61: gracefully shutting down
2022/08/03 17:00:48 [notice] 63#63: gracefully shutting down
2022/08/03 17:00:48 [notice] 64#64: gracefully shutting down
2022/08/03 17:00:48 [notice] 59#59: exiting
2022/08/03 17:00:48 [notice] 65#65: gracefully shutting down
2022/08/03 17:00:48 [notice] 62#62: exiting
2022/08/03 17:00:48 [notice] 60#60: exiting
2022/08/03 17:00:48 [notice] 66#66: gracefully shutting down
2022/08/03 17:00:48 [notice] 67#67: gracefully shutting down
2022/08/03 17:00:48 [notice] 63#63: exiting
2022/08/03 17:00:48 [notice] 68#68: gracefully shutting down
2022/08/03 17:00:48 [notice] 64#64: exiting
2022/08/03 17:00:48 [notice] 61#61: exiting
2022/08/03 17:00:48 [notice] 69#69: gracefully shutting down
2022/08/03 17:00:48 [notice] 65#65: exiting
2022/08/03 17:00:48 [notice] 66#66: exiting
2022/08/03 17:00:48 [notice] 71#71: gracefully shutting down
2022/08/03 17:00:48 [notice] 70#70: gracefully shutting down
2022/08/03 17:00:48 [notice] 67#67: exiting
...
2022/08/03 17:00:48 [notice] 65#65: exit
2022/08/03 17:00:48 [notice] 73#73: gracefully shutting down
...
2022/08/03 17:00:48 [notice] 74#74: exiting
2022/08/03 17:00:48 [notice] 83#83: gracefully shutting down
2022/08/03 17:00:48 [notice] 72#72: exiting
2022/08/03 17:00:48 [notice] 77#77: gracefully shutting down
2022/08/03 17:00:48 [notice] 77#77: exiting
2022/08/03 17:00:48 [notice] 77#77: exit
I0803 17:00:48.780547       1 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"nginx-ingress", Name:"nginx-config", UID:"961b1b89-3765-4eb8-9f5f-cfd8212012a8", APIVersion:"v1", ResourceVersion:"1612730", FieldPath:""}): type: 'Normal' reason: 'Updated' Configuration from nginx-ingress/nginx-config was updated 
I0803 17:00:48.780573       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"delivery-ingress", UID:"23f93b2d-c3c8-48eb-a2a1-e2ce0453677f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1527358", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/delivery-ingress was added or updated 
I0803 17:00:48.780585       1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated 
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 72
2022/08/03 17:00:48 [notice] 26#26: worker process 72 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 30
2022/08/03 17:00:48 [notice] 26#26: worker process 30 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 35 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 77 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 73
2022/08/03 17:00:48 [notice] 26#26: worker process 73 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 37
2022/08/03 17:00:48 [notice] 26#26: worker process 29 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 32 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 37 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 38 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 41 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 47 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 49 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 63 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 64 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 75 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 47
2022/08/03 17:00:48 [notice] 26#26: worker process 34 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 43 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 48 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 53 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 54 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 59 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 61 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 66 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 55
2022/08/03 17:00:48 [notice] 26#26: worker process 50 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 55 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 83
2022/08/03 17:00:48 [notice] 26#26: worker process 28 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 31 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 42 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 51 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 52 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 56 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 62 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 68 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 71 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 83 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 33
2022/08/03 17:00:48 [notice] 26#26: worker process 33 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 58
2022/08/03 17:00:48 [notice] 26#26: worker process 58 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 57
2022/08/03 17:00:48 [notice] 26#26: worker process 27 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 57 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 40
2022/08/03 17:00:48 [notice] 26#26: worker process 40 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 45 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 60 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 65 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 67 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 69 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 70 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 74 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 86 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received

I'm not sure what the issue is, and I can't figure out why it's working when I use different hosts, and not working when I try to use different paths.

I thought it could be resource limits, but I only have the requests, no limits. There is already a default IngressClass

I installed the ingress controller via manifests following the steps here

Update: To add the deployments running in the cluster.

# nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx-main
  template:
    metadata:
      labels:
        run: nginx-main
    spec:
      containers:
      - image: nginx
        name: nginx

# nginx-deploy-green.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: nginx
  name: nginx-deploy-green
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx-green
  template:
    metadata:
      labels:
        run: nginx-green
    spec:
      volumes:
      - name: webdata
        emptyDir: {}
      initContainers:
      - name: web-content
        image: busybox
        volumeMounts:
        - name: webdata
          mountPath: "/webdata"
        command: ["/bin/sh", "-c", 'echo "<h1>I am <font color=green>GREEN</font></h1>" > /webdata/index.html']
      containers:
      - image: nginx
        name: nginx
        volumeMounts:
        - name: webdata
          mountPath: "/usr/share/nginx/html"

# nginx-deploy-blue.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: nginx
  name: nginx-deploy-blue
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx-blue
  template:
    metadata:
      labels:
        run: nginx-blue
    spec:
      volumes:
      - name: webdata
        emptyDir: {}
      initContainers:
      - name: web-content
        image: busybox
        volumeMounts:
        - name: webdata
          mountPath: "/webdata"
        command: ["/bin/sh", "-c", 'echo "<h1>I am <font color=blue>BLUE</font></h1>" > /webdata/index.html']
      containers:
      - image: nginx
        name: nginx
        volumeMounts:
        - name: webdata
          mountPath: "/usr/share/nginx/html"

CodePudding user response：

I believe the reason is that the nginx service you have deployed for "blue" and "green" deployments are not serving anything on the /blue and /green endpoints. You need to create these endpoints inside nginx so that they become valid. Your ingress objects appear fine.

That's also why you see / working in all cases since by default nginx is hosting a web page on the path / but not on the paths /blue and /green. Simply creating the respective endpoints in your services for nginx should fix the problem.

CodePudding user response：

Based on the comments from zer0 answered, try:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-resource-3
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /  # <-- add
spec:
  ingressClassName: nginx
  ...

Page with different font color should response when your browse to http://nginx.example.com/blue or green