EKS Ingress with Single ALB, multiple namespaces, and External DNS-CodePudding

I'm trying to configure a single ALB across multiple namespaces in aws EKS, each namespace has its own ingress resource.

I'm trying to configure the ingress controller aws-loadbalancer-controller on a k8s v1.20.

The problem i'm facing is that each time I try to deploy a new service it always spin-up a new classic loadbalancer in addition to the shared ALB specified in the ingress config.

# service-realm1-dev.yaml:
apiVersion: v1
kind: Service
metadata:
  name: sentinel
  annotations:
    external-dns.alpha.kubernetes.io/hostname: realm1.dev.sentinel.mysite.io
  namespace: realm1-dev
  labels:
    run: sentinel
spec:
  ports:
    - port: 5001
      name: ps1
      protocol: TCP
  selector:
    app: sentinel
  type: LoadBalancer

# ingress realm1-app
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: sentinel-ingress
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
    alb.ingress.kubernetes.io/healthcheck-port: traffic-port
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: "15"
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "5"
    alb.ingress.kubernetes.io/success-codes: 200-300
    alb.ingress.kubernetes.io/healthy-threshold-count: "2"
    alb.ingress.kubernetes.io/unhealthy-threshold-count: "2"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}]'
  name: sentinel-ingress-controller
  namespace: realm1-dev
spec:
  rules:
    - host: realm1.dev.sentinel.mysite.io
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              servicePort: use-annotation
              serviceName: sentinel

Also I'm using external dns to create a route53 reecodset, and then I use the same configured DNS to route requests to the specific eks service, is there any issue with this approach ?

CodePudding user response：

Unfortunately the tool being used for your usecase is wrong. AWS Load Balancer Controller will create a new load balancer for every ingress resource and I think, it makes a network load balancer for every service resource.

For your use-case, the best option is to use nginx ingress controller. You can deploy the nginx ingress controller in any 1 namespace and then create ingress resources throughout your cluster and you can have path/hostname based routing across your cluster.

In case you have many teams/projects/applications and you want to avoid a single point of failure where all your apps depend on 1 ELB, you can deploy more than 1 nginx ingress controller in your k8s cluster.

You just need to define a ingress-class variable in your nginx ingress controller deployment and add that ingress-class annotation on your applications. This way, applications having ingress-class:nginxA annotation will be clustered with the nginx ingress controller that has ingress-class=nginxA in its deployment.

CodePudding user response：

I was able to make it work using only one ALB, @YYashwanth, Using Nginx was my fallback plan, I'm trying to make the configuration as simple as possible, maybe in the future when we will try to deploy our solution in other cloud providers we will use nginx ingress controller.

1- To start the service type should be node port, use loadbalancer will create a classic LB.

apiVersion: v1
kind: Service
metadata:
  name: sentinel-srv
  annotations:
    external-dns.alpha.kubernetes.io/hostname: operatorv2.dev.sentinel.marjory.io
  namespace: operatorv2-dev
  labels:
    run: jsflow-sentinel
spec:
  ports:
    - port: 80
      targetPort: 80
      name: ps1
      protocol: TCP
  selector:
    app: sentinel-app
  type: NodePort

2- Second we need to configure group.name, for the ingress controller to merge all ingress configurations using the annotation alb.ingress.kubernetes.io/group.name

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: "15"
    alb.ingress.kubernetes.io/healthcheck-path: /
    alb.ingress.kubernetes.io/healthcheck-port: traffic-port
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "5"
    alb.ingress.kubernetes.io/healthy-threshold-count: "2"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80} ]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/success-codes: "200"
    alb.ingress.kubernetes.io/tags: createdBy=aws-controller
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/unhealthy-threshold-count: "2"
    external-dns.alpha.kubernetes.io/hostname: operatorv2.dev.sentinel.marjory.io
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: sentinel-group
  name: dev-operatorv2-sentinel-ingress-controller
  namespace: operatorv2-dev
spec:
  rules:
    - host: operatorv2.dev.sentinel.mysite.io
      http:
        paths:
          - path: /*
            backend:
              servicePort: 80
              serviceName: sentinel-srv