Application Gateway (or Load Balancer) with App Services using Terraform?

Question

I am looking for the best way to implement load balancing (including redirection between multiple app services based on URL path) with App Services. This is what I have right now:

# App Service Plan
resource "azurerm_app_service_plan" "frontend" {
  name                = "${local.prefix}-frontend-asp"
  location            = azurerm_resource_group.frontend.location
  resource_group_name = azurerm_resource_group.frontend.name
  kind                = "Linux"
  reserved            = true
  app_service_environment_id = azurerm_app_service_environment_v3.frontend.id

  sku {
    tier = "Standard"
    size = "S1"
  }
}

# Environment
resource "azurerm_app_service_environment_v3" "frontend" {
  name                = "${local.prefix}-frontend-env"
  resource_group_name = azurerm_resource_group.frontend.name
  subnet_id           = azurerm_subnet.frontend.id

  cluster_setting {
    name  = "DisableTls1.0"
    value = "1"
  }

  cluster_setting {
    name  = "InternalEncryption"
    value = "true"
  }

  cluster_setting {
    name  = "FrontEndSSLCipherSuiteOrder"
    value = "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
  }

  tags = {
    env         = "production"
    terraformed = "true"
  }
}

# Main App Service
resource "azurerm_app_service" "frontend" {
  name                = "${local.prefix}-frontend-app"
  location            = azurerm_resource_group.frontend.location
  resource_group_name = azurerm_resource_group.frontend.name
  app_service_plan_id = azurerm_app_service_plan.frontend.id

  site_config {
    linux_fx_version = "NODE|14"
    app_command_line = "pm2 serve /home/site/wwwroot --no-daemon --spa"
  }

  app_settings = local.front_env_variables
}

[I am using app_service_environment_id = azurerm_app_service_environment_v3.frontend.id as the best way of having App Service inside a Subnet controlled by me, to be able to control basic security via Network Security Group. I found out that azurerm_app_service_virtual_network_swift_connection is not really what I need since it only allows App Service to talk to other resources inside a particular subnet, not necessairly does it place an App Service in the subnet per se. Correct me if I'm wrong.]

# Resource Group
resource "azurerm_resource_group" "networking" {
  name     = "${local.prefix}-networking-rg"
  location = local.location
  provider = azurerm.uat
}

# Virtual Network
resource "azurerm_virtual_network" "vnet" {
  name                = "${local.prefix}-vnet"
  location            = azurerm_resource_group.networking.location
  resource_group_name = azurerm_resource_group.networking.name
  address_space       = ["10.1.0.0/16"]

  tags = {
    environment = local.prefix
  }
}

# Subnets for App Service instances
resource "azurerm_subnet" "frontend" {
  name                 = "frontend-app"
  resource_group_name  = azurerm_resource_group.networking.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.1.1.0/24"]

  delegation {
    name = "delegation"
  
    service_delegation {
    name    = "Microsoft.Web/hostingEnvironments"
    actions = ["Microsoft.Network/virtualNetworks/subnets/action"]
    }
  }
}

# Route tables
resource "azurerm_route_table" "local" {
  name                          = "local"
  location            = azurerm_resource_group.networking.location
  resource_group_name = azurerm_resource_group.networking.name
  disable_bgp_route_propagation = false

  route {
    name           = "route1"
    address_prefix = "10.1.0.0/16"
    next_hop_type  = "vnetlocal"
  }

  tags = {
    environment = "uat"
  }
}

resource "azurerm_subnet_route_table_association" "frontend" {
  subnet_id      = azurerm_subnet.frontend.id
  route_table_id = azurerm_route_table.local.id
}

# Frontend App Security
resource "azurerm_network_security_group" "frontend" {
  name                = "${local.prefix}-frontend-sg"
  location            = azurerm_resource_group.frontend.location
  resource_group_name = azurerm_resource_group.frontend.name

  security_rule {
    name                       = "test123"
    priority                   = 100
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "*"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }
}

resource "azurerm_subnet_network_security_group_association" "frontend" {
  subnet_id                 = azurerm_subnet.frontend.id
  network_security_group_id = azurerm_network_security_group.frontend.id
}

[So what I am trying to do above is to make sure that App Service is in the private subnet (with no access from internet) and controlled by Security Group rule. I am coming from AWS world and this is the most logical setup to achieve it for me. Please correct me if I'm wrong again.]

Now, I want to have internet-facing Load Balancer (or Application Gateway) to gather the traffic based on the URL path or port, and then to route it to the correct App Service (which itself is in the private subnet). And I have no idea what's the best practice there.

This is the piece of code I have at the moment:

# Public IP
resource "azurerm_public_ip" "agw" {
  name                = "${local.prefix}-agw-pip"
  location              = azurerm_resource_group.agw.location
  resource_group_name   = azurerm_resource_group.agw.name
  allocation_method   = "Static"
  sku                 = "Standard"
  tags                = azurerm_resource_group.agw.tags
}

# Application Gateway
resource "azurerm_application_gateway" "agw" {
  name                = "${local.prefix}-agw"
  location              = azurerm_resource_group.agw.location
  resource_group_name   = azurerm_resource_group.agw.name

  sku {
    name     = "WAF_Medium"
    tier     = "WAF"
    capacity = 2
  }

  waf_configuration {
    enabled          = "true"
    firewall_mode    = "Detection"
    rule_set_type    = "OWASP"
    rule_set_version = "3.0"
  }

  gateway_ip_configuration {
    name      = "subnet"
    subnet_id = <to be created>
  }

  frontend_port {
    name = "http"
    port = 80
  }

  frontend_ip_configuration {
    name                 = "frontend"
    public_ip_address_id = "${azurerm_public_ip.agw.id}"
  }

  backend_address_pool {
    name        = "AppService"
    "fqdn_list" = ["${azurerm_app_service.frontend.name}.azurewebsites.net"]
  }

  http_listener {
    name                           = "http"
    frontend_ip_configuration_name = "frontend"
    frontend_port_name             = "http"
    protocol                       = "Http"
  }

  probe {
    name                = "probe"
    protocol            = "http"
    path                = "/"
    host                = "${azurerm_app_service.frontend.name}.azurewebsites.net"
    interval            = "30"
    timeout             = "30"
    unhealthy_threshold = "3"
  }

  backend_http_settings {
    name                  = "http"
    cookie_based_affinity = "Disabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 1
    probe_name            = "probe"
  }

  request_routing_rule {
    name                       = "http"
    rule_type                  = "Basic"
    http_listener_name         = "http"
    backend_address_pool_name  = "AppService"
    backend_http_settings_name = "http"
  }
}

But I have no idea how to "connect" this Application Gateway with the App Services, and if it's all right to have it along with App Service Environment.

score 0 · Answer 1 · answered Sep 16 '21 at 13:27

Your script seems to be fine except for

backend_address_pool {
    name        = "AppService"
    "fqdn_list" = ["${azurerm_app_service.frontend.name}.azurewebsites.net"]
  }

"fqdn_list" is not used now . it has been replaced by fqdns. So, you can use the below:

  backend_address_pool {
    name        = "AppService"
    fqdns = ["${azurerm_app_service.frontend.name}.azurewebsites.net"]
  }

 gateway_ip_configuration {
    name      = "subnet"
    subnet_id = <to be created>
  }
In this you have to create a subnet which will be used by your application gateway in the same vnet . So, you can use :
  gateway_ip_configuration {
    name      = "subnet"
    subnet_id = azurerm_subnet.appgw.id
  }

backend_http_settings {
    name                  = "http"
    cookie_based_affinity = "Disabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 1
    probe_name            = "probe"
  }

Add a feature flag "PickHostNameFromBackendAddress: true":

  backend_http_settings {
    name                  = "http"
    cookie_based_affinity = "Disabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 1
    probe_name            = "probe"
    pick_host_name_from_backend_address = true
  }

So, after this script is applied it creates the resources and you can see the backend pool in application gateway for your app service will be healthy.

But a better solution for your requirement is App gateway with private endpoint:

You can use the code below:

    provider "azurerm" {
    features{}
}

data "azurerm_resource_group" "rg" {
  name     = "yourresourcegroup"
}

# Virtual Network
resource "azurerm_virtual_network" "vnet" {
  name                = "ansumanapp-vnet"
  location            = data.azurerm_resource_group.rg.location
  resource_group_name = data.azurerm_resource_group.rg.name
  address_space       = ["10.4.0.0/16"]
}

# Subnets for App Service instances and app gateway
resource "azurerm_subnet" "appserv" {
  name                 = "frontend-app"
  resource_group_name  = data.azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.4.1.0/24"]
  enforce_private_link_endpoint_network_policies = true
  }

resource "azurerm_subnet" "appgw" {
  name                 = "appgw-subnet"
  resource_group_name  = data.azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.4.2.0/24"]
  enforce_private_link_endpoint_network_policies = true
  }

 
# App Service Plan
resource "azurerm_app_service_plan" "frontend" {
  name                = "ansuman-frontend-asp"
  location            = data.azurerm_resource_group.rg.location
  resource_group_name = data.azurerm_resource_group.rg.name
  kind                = "Linux"
  reserved            = true

  sku {
    tier = "Premium"
    size = "P1V2"
  }
}


# Main App Service
resource "azurerm_app_service" "frontend" {
  name                = "ansuman-frontend-app"
  location            = data.azurerm_resource_group.rg.location
  resource_group_name = data.azurerm_resource_group.rg.name
  app_service_plan_id = azurerm_app_service_plan.frontend.id

}
#private endpoint

resource "azurerm_private_endpoint" "example" {
  name                = "${azurerm_app_service.frontend.name}-endpoint"
  location            = data.azurerm_resource_group.rg.location
  resource_group_name = data.azurerm_resource_group.rg.name
  subnet_id           = azurerm_subnet.appserv.id
  

  private_service_connection {
    name                           = "${azurerm_app_service.frontend.name}-privateconnection"
    private_connection_resource_id = azurerm_app_service.frontend.id
    subresource_names = ["sites"]
    is_manual_connection = false
  }
}

# private DNS
resource "azurerm_private_dns_zone" "example" {
  name                = "privatelink.azurewebsites.net"
  resource_group_name = data.azurerm_resource_group.rg.name
}

#private DNS Link
resource "azurerm_private_dns_zone_virtual_network_link" "example" {
  name                  = "${azurerm_app_service.frontend.name}-dnslink"
  resource_group_name   = data.azurerm_resource_group.rg.name
  private_dns_zone_name = azurerm_private_dns_zone.example.name
  virtual_network_id    = azurerm_virtual_network.vnet.id
  registration_enabled = false
}

resource "azurerm_public_ip" "agw" {
  name                = "ansuman-agw-pip"
  location              = data.azurerm_resource_group.rg.location
  resource_group_name   = data.azurerm_resource_group.rg.name
  allocation_method   = "Dynamic"
  sku                 = "Basic"
}

# Application Gateway
resource "azurerm_application_gateway" "agw" {
  name                = "ansuman-agw"
  location              = data.azurerm_resource_group.rg.location
  resource_group_name   = data.azurerm_resource_group.rg.name

  sku {
    name     = "WAF_Medium"
    tier     = "WAF"
    capacity = 2
  }

  waf_configuration {
    enabled          = "true"
    firewall_mode    = "Detection"
    rule_set_type    = "OWASP"
    rule_set_version = "3.0"
  }

  gateway_ip_configuration {
    name      = "subnet"
    subnet_id = azurerm_subnet.appgw.id
  }

  frontend_port {
    name = "http"
    port = 80
  }

  frontend_ip_configuration {
    name                 = "frontend"
    public_ip_address_id = "${azurerm_public_ip.agw.id}"
  }

  backend_address_pool {
    name        = "AppService"
    fqdns = ["${azurerm_app_service.frontend.name}.azurewebsites.net"]
  }

  http_listener {
    name                           = "http"
    frontend_ip_configuration_name = "frontend"
    frontend_port_name             = "http"
    protocol                       = "Http"
  }

  probe {
    name                = "probe"
    protocol            = "http"
    path                = "/"
    host                = "${azurerm_app_service.frontend.name}.azurewebsites.net"
    interval            = "30"
    timeout             = "30"
    unhealthy_threshold = "3"
  }

  backend_http_settings {
    name                  = "http"
    cookie_based_affinity = "Disabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 1
    probe_name            = "probe"
    pick_host_name_from_backend_address = true
  }

  request_routing_rule {
    name                       = "http"
    rule_type                  = "Basic"
    http_listener_name         = "http"
    backend_address_pool_name  = "AppService"
    backend_http_settings_name = "http"

  }
}

Note: App service Sku should be Premium and tier can be as per your requirement for it to have access with private-endpoint.

Reference: Securing Azure Web Apps using Application Gateways and vNets

Thanks for the alternate solution!was able to do it from CLI but your answer helped me for doing it in terraform. — UserP, Sep 16 '21 at 17:31
One thing that is not perfect, is that my above setting is still reachable from the internet. How do I ensure that App Services are internal and not facing internet? — P09, Sep 17 '21 at 15:05
@P09, adding the app services to a private endpoint will make the app service restricted to internet and then adding it to the application gateway will make it accessible from app gateway ip. — Ansuman Bal, Sep 17 '21 at 15:14
Hello @P09, If the answer was helpful, Could you please [Accept it as an Answer](https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work), so that others who encounter the same issue can find this solution and fix their problem. — Ansuman Bal, Sep 23 '21 at 14:26
Hey @AnsumanBal-MT, I am trying to implement solution suggested by you, but we are getting "502 - Web server received an invalid response while acting as a gateway or proxy server" on an application gateway. I am thinking that maybe we are missing azurerm_private_link_service resource between app service and and private endpoint? — P09, Oct 06 '21 at 08:59
@P09, In place of azurerm_private_link resource , I have used as a private dns zone . I have followed the steps as mentioned in the link provided in answer and converted it to terraform .. will test it again and let you know — Ansuman Bal, Oct 06 '21 at 09:15
Thanks a lot @AnsumanBal-MT! I am almost getting it right now, trying to implement hostname based listening with 3 domains, will confirm how it goes. Just a side note, I found out that solution with ApGw and private endpoints destroyed our github deployments. Github is forbridden to deploy to app services. How can you overcome that? — P09, Oct 07 '21 at 10:57

Application Gateway (or Load Balancer) with App Services using Terraform?

1 Answers1