I'm trying to get the nginx ingress controller load balancer ip in Azure AKS. I figured I would use the kubernetes provider via:

data "kubernetes_service" "nginx_service" {
  metadata {
    name      = "${local.ingress_name}-ingress-nginx-controller"
    namespace = local.ingress_ns
  }

  depends_on = [helm_release.ingress]
}

However, i'm not seeing the IP address, this is what i get back:

nginx_service = [
        {
            cluster_ip                  = "10.0.165.249"
            external_ips                = []
            external_name               = ""
            external_traffic_policy     = "Local"
            health_check_node_port      = 31089
            load_balancer_ip            = ""
            load_balancer_source_ranges = []
            port                        = [
                {
                    name        = "http"
                    node_port   = 30784
                    port        = 80
                    protocol    = "TCP"
                    target_port = "http"
                },
                {
                    name        = "https"
                    node_port   = 32337
                    port        = 443
                    protocol    = "TCP"
                    target_port = "https"
                },
            ]
            publish_not_ready_addresses = false
            selector                    = {
                "app.kubernetes.io/component" = "controller"
                "app.kubernetes.io/instance"  = "nginx-ingress-internal"
                "app.kubernetes.io/name"      = "ingress-nginx"
            }
            session_affinity            = "None"
            type                        = "LoadBalancer"
        },
   ]

However when I pull down the service via kubectl I can get the IP address via:

 kubectl get svc nginx-ingress-internal-ingress-nginx-controller -n nginx-ingress -o json | jq -r '.status.loadBalancer.ingress[].ip'
10.141.100.158

Is this a limitation of kubernetes provider for AKS? If so, what is a workaround other people have used? My end goals is to use the IP to configure the application gateway backend.

I guess I can use local-exec, but that seem hacky. Howerver, this might be my only option at the moment.

Thanks,

Jerry

CodePudding user response：

although i strongly advise against creating resources inside Kubernetes with Terraform, you can do that:

Create a Public IP with Terraform -> Create the ingress-nginx inside Kubernetes with Terraform and pass annotations and loadBalancerIPwith data from your Terraform resources. The final manifest should look like this:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-resource-group: myResourceGroup
  name: ingress-nginx-controller
spec:
  loadBalancerIP: <YOUR_STATIC_IP>
  type: LoadBalancer

Terraform could look like this:

resource "kubernetes_service" "ingress_nginx" {
  metadata {
    name = "tingress-nginx-controller"
    
    annotations {
      "service.beta.kubernetes.io/azure-load-balancer-resource-group" = "${azurerm_resource_group.YOUR_RG.name}"
    }

  spec {
    selector = {
      app = <PLACEHOLDER>
    }
    port {
      port        = <PLACEHOLDER>
      target_port = <PLACEHOLDER>
    }

    type = "LoadBalancer"
    load_balancer_ip = "${azurerm_public_ip.YOUR_IP.ip_address}"
  }
}

CodePudding user response：

Unfortunately, this is for internal ingress and not public facing and the IP is allocated dynamically. We currently dont want to use static ips

This is what I came up with:

resource "null_resource" "kubectl" {
  provisioner "local-exec" {
    command = "kubectl get svc nginx-ingress-internal-ingress-nginx-controller -n nginx-ingress -o json | jq -r '.status.loadBalancer.ingress[].ip' --kubeconfig <(echo $KUBECONFIG | base64 --decode)" > ip
    interpreter = ["/bin/bash", "-c"]

    environment = {
      KUBECONFIG = base64encode(var.kubeconfig)
  }
}

data "local_file" "nginx-ip" {
  filename = "${path.module}/ip"
  depends_on = ["null_resource.kubectl"]
}

output nginx_ip {
  description = "IP address of the internal nginx controller
  value = data.local_file.nginx-ip.content
}