Azure AKS deployment raised an "InsufficientSubnetSize" error

Question

I'm trying to deploy an Azure AKS instance via ARM template.
I have a requirement to integrate the AKS instance into an existing Vnet.
I have a dedicated subnet for AKS service.
However, deployment has failed with the following error:

{"code":"DeploymentFailed","message":"At least one resource deployment operation  failed.  
Please list deployment operations for details. Please see  
https://aka.ms/arm-debug for usage details.","details":  
[{"code":"BadRequest","message":"{\r\n \"code\": \"InsufficientSubnetSize\",\r\n  
\"message\": \"Pre-allocated IPs 93 exceeds IPs available in Subnet 11\",\r\n  
\"target\": \"agentPoolProfile.count\"\r\n}"}]}

I'm using the following address space for Vnet: XX.XX.XX.0/24 (XX.XX.XX.0 - XX.XX.XX.255 which has 256 addresses.
I have a set of dedicated subnets within this Vnet, each of /28 mask (11+5 addresses depth):

XX.XX.XX.0/28  
XX.XX.XX.16/28  
XX.XX.XX.64/28  
XX.XX.XX.128/28  
XX.XX.XX.144/28  
XX.XX.XX.160/28  
XX.XX.XX.176/28

The subnet XX.XX.XX.144/28 is planned to be used in AKS.
The current AKS instance ARM template is as follows:

"resources": [
        {
            "type": "Microsoft.ContainerService/managedClusters",
            "apiVersion": "2019-04-01",
            "name": "[parameters('resourceName')]",
            "location": "[parameters('location')]",
            "dependsOn": [],
            "tags": {},
            "properties": {
                "kubernetesVersion": "[parameters('kubernetesVersion')]",
                "enableRBAC": "[parameters('enableRBAC')]",
                "dnsPrefix": "[parameters('dnsPrefix')]",
                "agentPoolProfiles": [
                    {
                        "name": "agentpool",
                        "osDiskSizeGB": "[parameters('osDiskSizeGB')]",
                        "count": "3",
                        "vmSize": "[parameters('agentVMSize')]",
                        "osType": "[parameters('osType')]",
                        "storageProfile": "ManagedDisks",
                        "maxPods": "30",
                        "vnetSubnetID": "/subscriptions/XXXXX/resourceGroups/XXXX/providers/Microsoft.Network/virtualNetworks/VNET_NAME/subnets/akssubnet"
                    }
                ],
                "servicePrincipalProfile": {
                    "ClientId": "[parameters('servicePrincipalClientId')]",
                    "Secret": "[parameters('servicePrincipalClientSecret')]"
                },
                "networkProfile": {
                    "networkPlugin": "azure",
                    "serviceCidr": "10.0.0.0/16",
                    "dnsServiceIP": "10.0.0.10",
                    "dockerBridgeCidr": "172.17.0.1/16"
                },
                "addonProfiles": {
                    "httpApplicationRouting": {
                        "enabled": "[parameters('enableHttpApplicationRouting')]"
                    },
                    "omsagent": {
                        "enabled": "[parameters('enableOmsAgent')]",
                        "config": {
                            "logAnalyticsWorkspaceResourceID": "[parameters('omsWorkspaceId')]"
                        }
                    }
                }
            }
        },        
            "subscriptionId": "[split(parameters('omsWorkspaceId'),'/')[2]]",
            "resourceGroup": "[split(parameters('omsWorkspaceId'),'/')[4]]"
        }
    ]

Network profile parameters were set according to the following article: Microsoft.ContainerService managedClusters template reference

A CIDR of 10.0.0.0/16 is of a private range and isn't interfering with my existing Vnet range.

I need advice on how to deal with this deployment error.

Upd:
I've tried the deployment with the values of my Vnet/subnets but stil it's failing:

Upd2:

Per MS documentation "Minimum number of pods on the initial cluster creation using Azure CNI type is 30" which leads to the following number of subnet range in my case according to the formula: (number of nodes + 1) + ((number of nodes + 1) * maximum pods per node that you configure) = (3+1) + ((3+1)*30) = 124

So the multiplier of 30 will be always present even if the number of pods is set to 1 in ARM template for example.

Upd3:

However, as I was unable to extend the existing subnet range I've managed to deploy the AKS instance using the following configuration:

"parameters": {
 "SvcCidr": {
      "type": "string",
      "defaultValue": "10.0.0.0/16",
      "metadata": {
        "description": "Maximum number of pods that can run on a node."
      }
    },
    "PodCidr": {
      "type": "string",
      "defaultValue": "10.244.0.0/16",
      "metadata": {
        "description": "Maximum number of pods that can run on a node."
      }
    },
    "DnsSvcIP": {
      "type": "string",
      "defaultValue": "10.0.0.10",
      "metadata": {
        "description": "Maximum number of pods that can run on a node."
      }
    },
    "DockerCidr": {
      "type": "string",
      "defaultValue": "",

"variables": {
    "vnetSubnetId": "[resourceId(resourceGroup().name, 'Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('vnetSubnetName'))]",

"resources": [
{
      "type": "Microsoft.ContainerService/managedClusters",
 "agentPoolProfiles": [
          {
      "vnetSubnetID": "[variables('vnetSubnetId')]",
 "networkProfile": {
          "networkPlugin": "[parameters('NetPlugin')]",
          "serviceCidr": "[parameters('SvcCidr')]",
          "podCidr": "[parameters('PodCidr')]",
          "DNSServiceIP": "[parameters('DnsSvcIP')]",
          "dockerBridgeCidr": "[parameters('DockerCidr')]"

Which leads to the provision of my subnet range IP addresses only to cluster nodes while the pods will use the private IP addresses range.

score 3 · Accepted Answer · answered Jun 24 '19 at 07:10

3

For your issue, when you use the azure module network, as it shows about the Calculation method in other answers, your subnet could just have one node. But actually, the IP address number of your subnet is not enough for just one node. Because there are already pods need the IP address when you create the AKS cluster in default, for example, the metric server and etc.

So you just can use the network nodule kubelet. In this module, just the node need to IP address in the subnet. And just use this network module, you can have 3 nodes as you want and use your existing subnet with just 8 IP address. For more details, see Use kubenet networking with your own IP address ranges in Azure Kubernetes Service (AKS).

answered Jun 24 '19 at 07:10

Charles Xu

29,862
2
22
39

If I understand you correctly you're advising to switch to kubenet instead of Azure CNI. However, kubenet doesn't support the Vnet integration which is a mandatory requirement in my case. Could you please confirm that using kubenet I'll be able to integrate the AKS cluster to my existing Vnet? – Sergey Jun 24 '19 at 07:24
@Sergey Why not? You can use the subnet Id just like you use the azure network module. – Charles Xu Jun 24 '19 at 07:31
As the "vnetSubnetID" parameter is omitted when the kubenet is used during the deployment, could you please advise which of the following values must be substituted with my existing Vnet subnet range - Pod CIDR or Service CIDR? – Sergey Jun 24 '19 at 07:58
@Sergey It will work even you change nothing. The pods will use the NAT method to communicate with the outside through the node IP. Just change the `networkPlugin` with the value `kubelet`. – Charles Xu Jun 24 '19 at 08:04
Please see my initial post updated. Thanks for your advice, I've switched to this approach due to restrictions. – Sergey Jun 26 '19 at 04:33
Probably you're right. I've selected your approach and the @4c74356b41's explanation was about how the things work under the hood. Changed my decision vice versa. – Sergey Jun 26 '19 at 06:10
@Sergey The Calculation that he said is right. But for you, the IP number of your existing subnet is not enough. So you just can use the kubelet network module. This is the final result. – Charles Xu Jun 26 '19 at 06:43

4c74356b41 · Answer 2 · 2019-06-24T05:51:40.113

taken from the docs:

Subnets:

Must be large enough to accommodate the nodes, pods, and all Kubernetes and Azure resources that might be provisioned in your cluster. For example, if you deploy an internal Azure Load Balancer, its front-end IPs are allocated from the cluster subnet, not public IPs. The subnet size should also take into account upgrade operations or future scaling needs. To calculate the minimum subnet size including an additional node for upgrade operations: (number of nodes + 1) + ((number of nodes + 1) * maximum pods per node that you configure)

Example for a 50 node cluster: (51) + (51 * 30 (default)) = 1,581 (/21 or larger)

Example for a 50 node cluster that also includes provision to scale up an additional 10 nodes: (61) + (61 * 30 (default)) = 1,891 (/21 or larger)

If you don't specify a maximum number of pods per node when you create your cluster, the maximum number of pods per node is set to 30. The minimum number of IP addresses required is based on that value. If you calculate your minimum IP address requirements on a different maximum value, see how to configure the maximum number of pods per node to set this value when you deploy your cluster.

this means for your case you need 30*4 + 4 = 124 ip addresses needed minumum for this to work, but keep in mind if you would want to add 4 node and upgrade it wouldn't work. if you would to want to scale to 5 nodes it wouldn't work. Also, what's the point if such small subnets? you don't pay for the subnet size, so making them reasonably big is not an issue

means you need /25, technically. 128-4 (reserved by azure) = 124 ;)

Reading: https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni#plan-ip-addressing-for-your-cluster

Could you please clarify - the value of "vnetSubnetID" and the value of "serviceCidr" should be within the same Vnet but belong to different subnets? Or "serviceCidr" must be a "part" of the "vnetSubnetID"? — Sergey, Jun 24 '19 at 06:08
no, none of these must overlap and serviceCidr is a virtual ip range, it doesn't have to be within a subnet (in fact its better that its not with any subnet range). vnetsubnetid - resourceId of the vnet kubernetes worker nodes get provision. servicecidr - virtual ip range for pods on kubernetes — 4c74356b41, Jun 24 '19 at 06:29
Great, thanks. Leaving my initial settings as is (vnetSubnetID XX.XX.XX.144/28, serviceCidr 10.0.0.0/16), according to your calculations above, with the parameter "maxPods" set to "1" (as it's within the range 1-110) the deployment should be fine. The number of IPs is 1*4+4 = 8 which matches to my /28 mask. But it fails, and the (same) error clearly states: during the deployment, 93 of IPs were pre-allocated but in the specified subnet only available number is 11. So the question is - why the number 93 is calculated if the number 8 must be? What is wrong with my config, I cannot understand. — Sergey, Jun 24 '19 at 06:34
your max pods is set to 30, and setting it to 1 makes no sense. you would be only able to have 1 container per node — 4c74356b41, Jun 24 '19 at 06:40

Azure AKS deployment raised an "InsufficientSubnetSize" error

2 Answers2