1

One of the security warnings I have in Azure Advisor is:

Authorized IP ranges should be defined on Kubernetes Services

Restrict access to the Kubernetes Service Management API by granting API access only to IP addresses in specific ranges. It is recommended to limit access to authorized IP ranges to ensure that only applications from allowed networks can access the cluster.

Can read all about it here:

https://learn.microsoft.com/en-us/azure/aks/api-server-authorized-ip-ranges

The command to update an existing AKS cluster is the following:

az aks update \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --api-server-authorized-ip-ranges  73.140.245.0/24

Or to remove any:

az aks update \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --api-server-authorized-ip-ranges ""

Anyway, with this enabled, my deployment pipeline timesout because it can't connect to AKS. I've added the IP address that pops when it says it can't connect and still isn't able to connect.

So basically I'm relegated to removing the IP ranges when I need to run the pipeline and then adding it back on when it is done. Obviously not an ideal way of handling this.

How should I resolve this and does Azure have a more elegant way of whitelisting the IP from the pipeline in AKS?

cjones
  • 8,384
  • 17
  • 81
  • 175

3 Answers3

2

Shiraz got me going in the right direction.

However, the only relevant part is this part:

https://learn.microsoft.com/en-us/azure/devops/organizations/security/allow-list-ip-url?view=azure-devops#azure-pipelines-agents

Relevant being that my build and deployment agents are Microsoft hosted. The article Shiraz linked to primarily deals with on-premises Azure DevOps:

Azure DevOps Services | Azure DevOps Server 2020 | Azure DevOps Server 2019 | TFS 2018 - TFS 2015

At any rate, this is the relevant documentation for Microsoft hosted agents:

https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#networking

Basically, two options:

  1. Downloading the weekly JSON Azure IP Ranges and then adding every IP for your geographic area under the AzureDevOps.{region} sections. So if you are in the CentralUS region, it is not sufficient to just add that region. You have to add all the regions in the US because it could potentially us a different region when using the build and deployment agents.

  2. Writing a simple program to parse the json file for you and gather all of the relevant IPs. The example the documentation uses:

using Newtonsoft.Json.Linq;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;

namespace WeeklyFileIPRanges
{
    class Program
    {
        // Path to the locally saved weekly file
        const string weeklyFilePath = @"C:\MyPath\ServiceTags_Public_20200504.json";

        static void Main(string[] args)
        {
            // United States geography has the following regions:
            // Central US, East US 2, East US, North Central US, 
            // South Central US, West Central US, West US, West US 2
            List<string> USGeographyRegions = new List<string>
            {
                "centralus",
                "eastus",
                "eastus2",
                "northcentralus",
                "southcentralus",
                "westcentralus",
                "westus",
                "westus2"
            };

            // Load the weekly file
            JObject weeklyFile = JObject.Parse(File.ReadAllText(weeklyFilePath));
            JArray values = (JArray)weeklyFile["values"];

            foreach (string region in USGeographyRegions)
            {
                string azureCloudRegion = $"AzureCloud.{region}";
                Console.WriteLine(azureCloudRegion);

                var ipList =
                    from v in values
                    where (string)v["name"] == azureCloudRegion
                    select v["properties"]["addressPrefixes"];

                foreach (var ip in ipList.Children())
                {
                    Console.WriteLine(ip);
                }
            }
        }
    }
}

Then to add the IPs you would use something like:

az aks update -g <resource_group> -n <aks_deployment_name> --api-server-authorized-ip-ranges [every,single,ip,address,for,the,geographic,area]

If anyone has a better way, I'll gladly accept that as the answer. There has to be one given that that is +1600 IPs for the US geo area and --api-server-authorized-ip-ranges will only take a maximum of 200.

At that point, it is just easier to erase the authorized IP addresses temporarily.

There is some promising stuff here though:

How to get the IP Address for Azure DevOps Hosted Agents to add to the white list

Tom Nijhof
  • 542
  • 4
  • 11
cjones
  • 8,384
  • 17
  • 81
  • 175
  • Great! Thanks for sharing your solution here, you could accept it as the answer, so it could help other community members who get the same issues and we could archive this thread, thanks. – Joy Jan 01 '21 at 07:16
  • You will need to the AzureDevOps not AzureCloud. This will also limit the number of IPs you need to 1. Solving the maximum of 200 problem – Tom Nijhof Sep 03 '21 at 10:56
1

The IP ranges that you need to open for are listed here:

https://learn.microsoft.com/en-us/azure/devops/organizations/security/allow-list-ip-url?view=azure-devops

I did not find the IP address that you added in that list.

Shiraz Bhaiji
  • 64,065
  • 34
  • 143
  • 252
0

I have been using the below solution to add the Azure DevOps IP addresses to Authorized IP ranges for Azure Kubernetes Service. You can modify this solution to use it for any of the Azure Services.

You need to add two "Azure Cli" tasks - one to add Azure DevOps Agent IP address and another to remove the Azure DevOps Agent IP Address.

Add the first task before the Kubernetes tasks:

I added a new task in Azure DevOps pipeline with "Azure Cli" and added the below commands as inline script:

echo "Get Azure DevOps IP address"

azdoip=`curl -s icanhazip.com`

echo "Azure DevOps IP Address: $azdoip"

echo "Set Azure Subscription to MYSUBSCRIPTION"

az account set  --subscription "MYSUBSCRIPTION"

echo "Get credentials for AKS Cluster Admin"

az aks get-credentials --resource-group MYAKSRG --name MYAKSCLUSTER --admin --file  ~/.kube/config

Echo "Get existing authorized ip ranges"

authorizedips=`az aks show  --resource-group MYAKSRG  --name MYAKSCLUSTER  --query apiServerAccessProfile |jq -r '.authorizedIpRanges | join(",")'`

echo "Update Azure DevOps IP Address in AKS Cluster Authorized IP Ranges"

az aks update  --resource-group MYAKSRG  --name MYAKSCLUSTER  --api-server-authorized-ip-ranges $authorizedips,$azdoip

Once all the kubernetes tasks has been finished, add another "Azure Cli" task at the end to remove Azure DevOps IP.

echo "Set Azure Subscription to MYSUBSCRIPTION"

az account set  --subscription "MYSUBSCRIPTION"

echo "Get credentials for AKS Cluster Admin"

az aks get-credentials --resource-group MYAKSRG --name MYAKSCLUSTER --admin --file  ~/.kube/config

echo "Get New Authorized IP Ranges"

newauthorizedips=`az aks show  --resource-group MYAKSRG   --name MYAKSCLUSTER  --query apiServerAccessProfile |jq -r '.authorizedIpRanges | join(",")'`

echo "Remove AzDo IP and store it as a variable" #Removes last element from the array of IPs

authorizedips=`echo $newauthorizedips | awk 'BEGIN{FS=OFS=","}NF--'`

echo "Update AKS by restoring Authorized IPs"

az aks update  --resource-group MYAKSRG   --name MYAKSCLUSTER --api-server-authorized-ip-ranges $authorizedips
srsn
  • 175
  • 11