Strange behavior of the Istio Gateway port-CodePudding

I have a hard time understand how exactly is the Istio Gateway port used. I am referring to line 14 in the below example

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 8169
        name: http-test1
        protocol: HTTP
      hosts:
        - '*'

From the Istio documentation: The Port on which the proxy should listen for incoming connections. So indeed if you apply the above yaml file and check the istio-ingressgateway pod for listening TCP ports you will find that the port 8169 is actually used (see below output)

kubectl -n=istio-system exec istio-ingressgateway-8577c57fb6-p8zl5 -- ss -nl | grep 8169
tcp   LISTEN 0      4096               0.0.0.0:8169        0.0.0.0:*

But here comes the tricky part. If before you apply the Gateway you change the istio-ingressgateway service as follow:

apiVersion: v1
kind: Service
metadata:
  name: istio-ingressgateway
...
  - name: http5
    nodePort: 31169
    port: 8169
    protocol: TCP
    targetPort: 8069
...

And then you apply the Gateway then the actual port used is not 8169 but 8069. It seems like that the Gateway resource will check first for a matching port in the istio-ingressgateway service and use the targetPort of the service instead

kubectl -n=istio-system exec istio-ingressgateway-8577c57fb6-p8zl5 -- ss -nl | grep 8169
<empty result>
kubectl -n=istio-system exec istio-ingressgateway-8577c57fb6-p8zl5 -- ss -nl | grep 8069
tcp   LISTEN 0      4096               0.0.0.0:8069        0.0.0.0:*

Can anybody explain why? Thank you in advance for any help

CodePudding user response：

As far as I know, the gateways are virtual, you can define multiple gateways and expose different ports. However, these ports still need to be open on the istio-ingressgateway

So when you manually change the ports config on the actual ingressgateway, it makes sense that only that specific port is open after applying it. You are checking the open port on the ingressgateway and not on the virtual gateway.

Also, I don't think it is encouraged to directly edit the istio-ingressgateway service. If you want to customize the ingressgateway you can define a IstioOperator and apply it on installing Istio.

CodePudding user response：

You encountered an interesting aspect of Istio - how to configure Istio to expose a service outside of the service mesh using an Istio Gateway.

First of all, please note that the gateway configuration will be applied to the proxy running on a Pod (in your example on a Pod with labels istio: ingressgateway). Istio is responsible for configuring the proxy to listen on these ports, however it is the user's responsibility to ensure that external traffic to these ports are allowed into the mesh.

Let me show you with an example. What you encountered is expected behaviour, because that is exactly how Istio works.

First, I created a simple Gateway configuration (for the sake of simplicity I omit Virtual Service and Destination Rule configurations) like below:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 9091
      name: http-test-1
      protocol: HTTP
    hosts:
    - '*'

Then:

    $ kubectl apply -f gw.yaml
    gateway.networking.istio.io/gateway created

Let's check if our proxy is listening on port 9091. We can check it directly from the istio-ingressgateway-* pod or we can use the istioctl proxy-config listener command to retrieve information about listener configuration for the Envoy instance in the specified Pod:

    $ kubectl exec istio-ingressgateway-8c48d875-lzsng -n istio-system -- ss -tulpn | grep 9091
    tcp     LISTEN   0        1024             0.0.0.0:9091           0.0.0.0:*      users:(("envoy",pid=14,fd=35))
    
    $ istioctl proxy-config listener istio-ingressgateway-8c48d875-lzsng  -n istio-system
    ADDRESS PORT  MATCH DESTINATION
    0.0.0.0 9091  ALL   Route: http.9091

Exposing this port on the pod doesn't mean that we are able to reach it from the outside world, but it is possible to reach this port internally from another pod:

    $ kubectl get pod -n istio-system -o wide
    NAME                                  READY   STATUS    RESTARTS   AGE   IP         
    istio-ingressgateway-8c48d875-lzsng   1/1     Running   0          43m   10.4.0.4
    
    $ kubectl exec -it test -- curl 10.4.0.4:9091
    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    ...

To make it accessible externally we need to expose this port on istio-ingressgateway Service:

    ...
      ports:
      - name: http-test-1
        nodePort: 30017
        port: 9091
        protocol: TCP
        targetPort: 9091
    ...

After this modification, we can reach port 9091 from the outside world:

    $ curl http://<PUBLIC_IP>:9091
    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    ...

Please note that nothing has changed from Pod's perspective:

    $ kubectl exec istio-ingressgateway-8c48d875-lzsng -n istio-system -- ss -tulpn | grep 9091
    tcp     LISTEN   0        1024             0.0.0.0:9091           0.0.0.0:*      users:(("envoy",pid=14,fd=35))
    
    $ istioctl proxy-config listener istio-ingressgateway-8c48d875-lzsng  -n istio-system
    ADDRESS PORT  MATCH DESTINATION
    0.0.0.0 9091  ALL   Route: http.9091

Now let's change the targetPort: 9091 to targetPort: 9092 in the istio-ingressgateway Service configuration and see what happens:

    ...
      ports:
      - name: http-test-1
        nodePort: 30017
        port: 9091
        protocol: TCP
        targetPort: 9092   <--- "9091" to "9092"
    ...
    
    $ kubectl exec istio-ingressgateway-8c48d875-lzsng -n istio-system -- ss -tulpn | grep 9091
    tcp     LISTEN   0        1024             0.0.0.0:9091           0.0.0.0:*      users:(("envoy",pid=14,fd=35))
    
    $ istioctl proxy-config listener istio-ingressgateway-8c48d875-lzsng  -n istio-system
    ADDRESS PORT  MATCH DESTINATION
    0.0.0.0 9091  ALL   Route: http.9091

As you can see, it seems that nothing has changed from the Pod's perspective so far, but we also need to re-apply the Gateway configuration:

    $ kubectl delete -f gw.yaml && kubectl apply -f gw.yaml
    gateway.networking.istio.io "gateway" deleted
    gateway.networking.istio.io/gateway created
    
    $ kubectl exec istio-ingressgateway-8c48d875-lzsng -n istio-system -- ss -tulpn | grep 9092
    tcp     LISTEN   0        1024             0.0.0.0:9092           0.0.0.0:*      users:(("envoy",pid=14,fd=35))
    
    $ istioctl proxy-config listener istio-ingressgateway-8c48d875-lzsng  -n istio-system
    ADDRESS PORT  MATCH DESTINATION
    0.0.0.0 9092  ALL   Route: http.9092

Our proxy is now listening on port 9092 (targetPort), but we can still reach port 9091 from the outisde as long as our Gateway specifies this port and it is open on the istio-ingressgateway Service.

    $ kubectl describe gw gateway -n istio-system | grep -A 4 "Port"
        Port:
          Name:      http-test-1
          Number:    9091
          Protocol:  HTTP
          
    $ kubectl get svc -n istio-system -oyaml | grep -C 2 9091
        - name: http-test-1
          nodePort: 30017
          port: 9091
          protocol: TCP
          targetPort: 9092
          
    $ curl http://<PUBLIC_IP>:9091
    <!DOCTYPE html>
    <html>
    <head>
    <title>Welcome to nginx!</title>
    ...