I am trying to implement AKS Baselines with terraform, but I can't get my Application Gateway connect to the internal load balancer created by AKS.

My AKS config contains of a solr instance and a service with azure-load-balancer-internal annotation. AKS and created LB are in the same SUBNET while Application Gateway has it's own SUBNET, but they are all in the same VNET.

Kubernetes.tf

resource "kubernetes_service" "solr-service" {
  metadata {
    name        = local.solr.name
    annotations = {
      "service.beta.kubernetes.io/azure-load-balancer-internal" : "true"
      "service.beta.kubernetes.io/azure-load-balancer-internal-subnet" : "aks-subnet"

    }
  }

  spec {
    external_traffic_policy = "Local"
    selector = {
      app = kubernetes_deployment.solr.metadata.0.labels.app
    }

    port {
      name        = "http"
      port        = 80
      target_port = 8983
    }

    type             = "LoadBalancer"
    load_balancer_ip = "192.168.1.200"
  }
}

This config creates an internal load balancer in the MC_* resource group with frontend IP 192.168.1.200. The health check in the metrics blade is returning 100. So it looks like the created internal loadbalancer is working as expected.

Now I am trying to add this load balancer as backend_pool target in my Application gateway.

application-gateway.tf

resource "azurerm_application_gateway" "agw" {
  name                = local.naming.agw_name
  resource_group_name = azurerm_resource_group.this.name
  location            = azurerm_resource_group.this.location

  sku {
    name     = "Standard_Medium"
    tier     = "Standard"
    capacity = 1
  }

  gateway_ip_configuration {
    name      = "Gateway-IP-Config"
    subnet_id = azurerm_subnet.agw_snet.id
  }

  frontend_port {
    name = "http-port"
    port = 80
  }

  frontend_ip_configuration {
    name                 = "public-ip"
    public_ip_address_id = azurerm_public_ip.agw_ip.id
  }

  backend_address_pool {
    name         = "lb"
    ip_addresses = ["192.168.1.200"]
  }

  backend_http_settings {
    name                  = "settings"
    cookie_based_affinity = "Disabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 60
  }

  http_listener {
    name                           = "http-listener"
    frontend_ip_configuration_name = "public-ip"
    frontend_port_name             = "http-port"
    protocol                       = "Http"
  }

  request_routing_rule {
    name                       = local.request_routing_rule_name
    rule_type                  = "Basic"
    http_listener_name         = "http-listener"
    backend_address_pool_name  = "lb"
    backend_http_settings_name = "settings"
  }
}

I would expect Application Gateway now be connected to the internal load balancer and send all request over to it. But I get the message, that all backend pools are unhealthy. So it looks like, the Gateway can't access the provided IP.

I took a look at the Azure GIT baseline, but as far as I can see, they using FQDN instead of IP. I am pretty sure it's just some minor configuration issue, but I just can't find it.

I tried already using the Application Gateway as ingress controller (or http routing) and this worked, but I would like to implement it with internal load balancer, I also tried to add health check to the backend nodepool, this did not worked.

EDIT: I changed the LB to public and added the public IP to the Application Gateway and everything worked, so it looks like this is the issue, but I don't get why Application Gateway can't access the sibling subnet. I don't have any restrictions in place and by default Azure allows communication between subnets.

CodePudding user response：

My mistake was to place the internal-load-balancer into the same snet like my kubernetes. When I changed the code and provided its own subnet, everything worked out fine. My final service config:

resource "kubernetes_service" "solr-service" {
  metadata {
    name        = local.solr.name
    annotations = {
      "service.beta.kubernetes.io/azure-load-balancer-internal" : "true"
      "service.beta.kubernetes.io/azure-load-balancer-internal-subnet" : "lb-subnet"
    }
  }

  spec {
    external_traffic_policy = "Local"
    selector = {
      app = kubernetes_deployment.solr.metadata.0.labels.app
    }

    port {
      name        = "http"
      port        = 80
      target_port = 8983
    }

    type             = "LoadBalancer"
    load_balancer_ip = "192.168.3.200"
  }
}