I'm testing Gateway API with GKE (version 1.21.11-gke.1100). I'm using gatewayClassName: gke-l7-rilb
for a Gateway with TLS between the client and the gateway. HTTPS is working perfectly between the client and the loadbalancer using a managed regional SSL certificate .
I have 2 Httproutes referencing 2 kube services (backendRefs
). One service is accessible through HTTP and the other through HTTPS (argo-server service from Argo Workflows project if it may helps).
When I create the httproute referencing the service using HTTP, the GCP load balancer backend service is created and working without any problem (Healthy).
But when I create the httproute referencing argo-service, a GCP load balancer backend service is created but not working (not healthy) with an endpoint protocol set to HTTP rather than HTTPS. You should know that I made sure to add to argo-server service the annotation cloud.google.com/app-protocols: '{"web":"HTTPS"}'
to enable HTTPS between the load balancer and argo-server application.
If I create the same geatway api configuration using ingress resource and the same argo service definition, endpoint protocol ( of the GCP load balancer backend service )is set correctly to HTTPS and is perfectly healthy and working.
As if httproute of the gateway API the GKE gateway controller are not taking into consideration the cloud.google.com/app-protocols
service annotation although it is mentioned that it's relevant for the gateway API here.
Edit 1: added yaml files
- Gateway:
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"Gateway","metadata":{"annotations":{},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"regional-internal-https","namespace":"exposition"},"spec":{"addresses":[{"type":"NamedAddress","value":"dev-gateway-internal-lb-static-ip"}],"gatewayClassName":"gke-l7-rilb","listeners":[{"allowedRoutes":{"kinds":[{"kind":"HTTPRoute"}],"namespaces":{"from":"Selector","selector":{"matchLabels":{"exposed":"true"}}}},"name":"https","port":443,"protocol":"HTTPS","tls":{"mode":"Terminate","options":{"networking.gke.io/pre-shared-certs":"plat-dev-europe-west1"}}}]}}
networking.gke.io/addresses: ""
networking.gke.io/backend-services: gkegw1-bkib-argo-argo-server-2746-8ktcvo8d0ktp,
gkegw1-bkib-demo-application-demo-service-80-y5bgcnm71kjv, gkegw1-bkib-exposition-gw-serve404-80-pciznuyt569p
networking.gke.io/firewalls: ""
networking.gke.io/forwarding-rules: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
networking.gke.io/health-checks: gkegw1-bkib-argo-argo-server-2746-8ktcvo8d0ktp,
gkegw1-bkib-demo-application-demo-service-80-y5bgcnm71kjv, gkegw1-bkib-exposition-gw-serve404-80-pciznuyt569p
networking.gke.io/last-reconcile-time: "2022-06-16T15:57:45Z"
networking.gke.io/ssl-certificates: ""
networking.gke.io/target-proxies: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
networking.gke.io/url-maps: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
creationTimestamp: "2022-06-15T08:28:20Z"
finalizers:
- gateway.finalizer.networking.gke.io
generation: 1
labels:
app.kubernetes.io/managed-by: gcp-cloud-build-deploy
managedFields:
- apiVersion: gateway.networking.k8s.io/v1alpha2
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:app.kubernetes.io/managed-by: {}
f:spec:
.: {}
f:addresses: {}
f:gatewayClassName: {}
f:listeners:
.: {}
k:{"name":"https"}:
.: {}
f:allowedRoutes:
.: {}
f:kinds: {}
f:namespaces:
.: {}
f:from: {}
f:selector:
.: {}
f:matchLabels:
.: {}
f:exposed: {}
f:name: {}
f:port: {}
f:protocol: {}
f:tls:
.: {}
f:mode: {}
f:options:
.: {}
f:networking.gke.io/pre-shared-certs: {}
manager: kubectl-client-side-apply
operation: Update
time: "2022-06-15T08:28:20Z"
- apiVersion: gateway.networking.k8s.io/v1alpha2
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:networking.gke.io/addresses: {}
f:networking.gke.io/backend-services: {}
f:networking.gke.io/firewalls: {}
f:networking.gke.io/forwarding-rules: {}
f:networking.gke.io/health-checks: {}
f:networking.gke.io/last-reconcile-time: {}
f:networking.gke.io/ssl-certificates: {}
f:networking.gke.io/target-proxies: {}
f:networking.gke.io/url-maps: {}
f:finalizers:
.: {}
v:"gateway.finalizer.networking.gke.io": {}
f:status:
f:addresses: {}
manager: GoogleGKEGatewayController
operation: Update
time: "2022-06-15T08:30:16Z"
name: regional-internal-https
namespace: exposition
resourceVersion: "42337844"
uid: 59333aea-1a79-4e9b-afbc-595ae9ccdfd7
spec:
addresses:
- type: NamedAddress
value: dev-gateway-internal-lb-static-ip
gatewayClassName: gke-l7-rilb
listeners:
- allowedRoutes:
kinds:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespaces:
from: Selector
selector:
matchLabels:
exposed: "true"
name: https
port: 443
protocol: HTTPS
tls:
mode: Terminate
options:
networking.gke.io/pre-shared-certs: plat-dev-europe-west1
status:
addresses:
- type: IPAddress
value: 10.163.112.28
conditions:
- lastTransitionTime: "1970-01-01T00:00:00Z"
message: Waiting for controller
reason: NotReconciled
status: Unknown
type: Scheduled
- Httproute:
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"HTTPRoute","metadata":{"annotations":{},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"argo-server","namespace":"argo"},"spec":{"hostnames":["argo-server.plat.dev.df.gcp.corp.modified.com"],"parentRefs":[{"kind":"Gateway","name":"regional-internal-https","namespace":"exposition"}],"rules":[{"backendRefs":[{"name":"argo-server","port":2746}]}]}}
creationTimestamp: "2022-06-15T12:27:04Z"
generation: 1
labels:
app.kubernetes.io/managed-by: gcp-cloud-build-deploy
managedFields:
- apiVersion: gateway.networking.k8s.io/v1alpha2
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:app.kubernetes.io/managed-by: {}
f:spec:
.: {}
f:hostnames: {}
f:parentRefs: {}
f:rules: {}
manager: kubectl-client-side-apply
operation: Update
time: "2022-06-15T12:27:04Z"
- apiVersion: gateway.networking.k8s.io/v1alpha2
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:parents: {}
manager: GoogleGKEGatewayController
operation: Update
time: "2022-06-15T12:29:02Z"
name: argo-server
namespace: argo
resourceVersion: "42362026"
uid: 981ce997-c574-4878-bec1-b03c7707838c
spec:
hostnames:
- argo-server.plat.dev.df.gcp.corp.modified.com
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: regional-internal-https
namespace: exposition
rules:
- backendRefs:
- group: ""
kind: Service
name: argo-server
port: 2746
weight: 1
matches:
- path:
type: PathPrefix
value: /
status:
parents:
- conditions:
- lastTransitionTime: "2022-06-16T17:00:11Z"
message: ""
reason: RouteAccepted
status: "True"
type: Accepted
- lastTransitionTime: "2022-06-16T17:00:11Z"
message: ""
reason: ReconciliationSucceeded
status: "True"
type: Reconciled
controllerName: networking.gke.io/gateway
parentRef:
group: gateway.networking.k8s.io
kind: Gateway
name: regional-internal-https
namespace: exposition
- Service:
apiVersion: v1
kind: Service
metadata:
annotations:
cloud.google.com/app-protocols: '{"web":"HTTPS"}'
cloud.google.com/backend-config: '{"default": "argo-server-backendconfig"}'
cloud.google.com/neg: '{"exposed_ports":{"2746":{}}}'
cloud.google.com/neg-status: '{"network_endpoint_groups":{"2746":"k8s1-f83345f9-argo-argo-server-2746-4d39c835"},"zones":["europe-west1-c"]}'
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/app-protocols":"{\"web\":\"HTTPS\"}","cloud.google.com/backend-config":"{\"default\": \"argo-server-backendconfig\"}","cloud.google.com/neg":"{\"ingress\": true}","cluster-autoscaler.kubernetes.io/safe-to-evict":"true"},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"argo-server","namespace":"argo"},"spec":{"ports":[{"name":"web","port":2746,"targetPort":2746}],"selector":{"app":"argo-server"}}}
creationTimestamp: "2022-06-15T11:44:07Z"
labels:
app.kubernetes.io/managed-by: gcp-cloud-build-deploy
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:cloud.google.com/app-protocols: {}
f:cloud.google.com/backend-config: {}
f:cluster-autoscaler.kubernetes.io/safe-to-evict: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:app.kubernetes.io/managed-by: {}
f:spec:
f:ports:
.: {}
k:{"port":2746,"protocol":"TCP"}:
.: {}
f:name: {}
f:port: {}
f:protocol: {}
f:targetPort: {}
f:selector:
.: {}
f:app: {}
f:sessionAffinity: {}
f:type: {}
manager: kubectl-client-side-apply
operation: Update
time: "2022-06-15T12:27:23Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:cloud.google.com/neg: {}
manager: GoogleGKEGatewayController
operation: Update
time: "2022-06-15T12:28:06Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:cloud.google.com/neg-status: {}
manager: glbc
operation: Update
time: "2022-06-15T12:28:06Z"
name: argo-server
namespace: argo
resourceVersion: "41692832"
uid: 25024d53-1d31-4165-8033-1843ec5d72ec
spec:
clusterIP: 10.163.247.121
clusterIPs:
- 10.163.247.121
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: web
port: 2746
protocol: TCP
targetPort: 2746
selector:
app: argo-server
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
CodePudding user response:
That annotation is not supported by the gke-l7-rilb GatewayClass according to the documentation.
Can you share the YAML files please ?
CodePudding user response:
I found a solution which I think of as a workaround.
Use
networking.gke.io/app-protocols: '{"web":"HTTPS"}'
annotation rather thancloud.google.com/app-protocols: '{"web":"HTTPS"}'
. This annotation is to use at service level, whereweb
is name of the port. This will enable HTTPS between the load balancer and the application (Endpoint protocol of the backend service created for the specified HTTPRoute). This is working perfectly withgatewayClassName: gke-l7-rilb
a Regional Internal Load Balancer.create a custom health check using
cloud.google.com/v1 BackendConfig
where you set the type to HTTPS and port to 2746. More details here https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#direct_health With ingresses, GCE ingress controller create automatically this health check from the application readiness probe but apparently this function is not yet implemented in the GKE Gateway controller.Make sure you have firewall rules allowing ingress traffic for Google Cloud health checks on 2746 port With ingresses, GCE ingress controller create automatically the required firewall rule but apparently this function is not yet implemented in the GKE Gateway controller.
Finally I said this is a workaround because I imagine and hope that future version of the GKE gateway controller will fix the 3 issues or points I mentioned above.