6

I'm trying to deploy an HDInsight cluster using an ARM template via Visual Studio. I've created an Azure Resource Group project in Visual Studio 2015, and added my resource definitions to the template JSON files.

However when I've gone to deploy it (by right-clicking the project, choosing Deploy -> New Deployment, entering my parameters), the output of Visual Studio shows (I've snipped out some boring stuff):

17:19:23 - Build started.

17:19:23 - Project "LaunchHdInsightCluster.deployproj" (StageArtifacts target(s)):

[snip]

17:20:27 - [VERBOSE] 17:20:27 - Resource Microsoft.HDInsight/clusters 'groupbhdinsight' provisioning status is running

17:31:06 - [ERROR] New-AzureRmResourceGroupDeployment : ExpiredAuthenticationToken: The access token expiry UTC time '3/14/2016 5:31:06 PM' is earlier than current UTC time '3/14/2016 5:31:07 PM'.

Note that the deploy only ran for 12 minutes before the access token expired - obviously for deploying an HDInsight cluster this is a problem (takes on average 20 minutes).

I'm just trying to understand what's going on under the hood here, as I can't find documentation for this. i.e:

What creates the access token and how? How long does it last for? I wasn't asked for any Azure creds when deploying - I'm assuming it must be the fact that I'm signed into Visual Studio using the same account I use in Azure, and it 'borrows' the authentication session, but this is just a guess

What determines the expiry time of the access token so I can prevent this happening again?

How do I refresh my authentication token?

James Allen
  • 6,406
  • 8
  • 50
  • 83
  • I have exactly the same issue. I triggered my deployment about twenty minutes ago (around 3/14/2016 6:10 PM UTC). Do you think it's a transient issue with Azure Resource Group? P.S. Region = West US – Igor Soloydenko Mar 14 '16 at 18:19
  • I don't see any Azure services being down: https://azure.microsoft.com/en-us/status/ There's one entry for this time window related to Classic VMs. Not sure if it's relevant: > SUMMARY OF IMPACT: Between 01:30 UTC on 10 Mar 2016 and 03:53 UTC on 14 Mar 2016 customers using the New Portal (portal.azure.com) were unable to add endpoints to Classic VMs. The same function was available in the Classic Portal (manage.windowsazure.com). – Igor Soloydenko Mar 14 '16 at 18:24
  • Well happily my deployment succeeded despite the error - I seem to have a fully working HDInsight cluster. I guess because the authentication timed out after the ARM template was sent to Azure, so the only 'problem' was not getting back a result (i.e. deployment succeeded). So I think the issue is purely a client side one with Visual Studio – James Allen Mar 14 '16 at 19:22
  • I found the following near the start of a powershell script called Deploy-AzureResourceGroup.ps1 which VS generated as part of the project template: `[Microsoft.Azure.Common.Authentication.AzureSession]::ClientFactory.AddUserAgent("VSAzureTools-$UI$($host.name)".replace(" ","_"), "2.8")`. I think VS calls this script to do the deployment. It looks like this grabs the token from VS. I might try changing this, e.g. to do Login-AzureRmAccount instead and do the login interactively – James Allen Mar 14 '16 at 19:43

5 Answers5

8

What's happening here is that the Azure Resource Group deployment in VS uses the PowerShell Script in the project to do deployment (even though the output is hosted in VS, we use that PS script to do the work). The PowerShell script is authenticated by using the token from your VS sign in. That token is only good for an hour and then VS will refresh it. Once it's handed off to PowerShell though, PowerShell doesn't automatically refresh it. So if you have the token for 59 minutes, it's going to expire soon after you start the deployment. The token could last for an hour, or anything less than that. We're working on a fix for this (i.e. have PowerShell automatically refresh the token) but that's a month or so out yet. See: https://github.com/Azure/azure-powershell/issues/1068

Workarounds: Unfortunately there's no good work around from VS. But...

  • As observed the deployment will continue just fine in Azure, it's just that VS/PS can no longer poll for status. You can monitor the deployment via the portal or PowerShell.
  • If you drop to PowerShell and run the script, PowerShell will automatically refresh the token when you login with credentials - you can get the exact command that VS runs by sifting through the output window - this doc also gives an overview of running the script manually: https://azure.microsoft.com/en-us/documentation/articles/vs-azure-tools-resource-groups-how-script-works/

Hope that helps...

bmoore-msft
  • 8,376
  • 20
  • 22
  • Thanks for the detailed answer @bmoore-msft! – James Allen May 16 '16 at 17:24
  • I dont know if you're still monitoring this thread, but i'm still getting it over a year later. In my instance, I've got a template with issues, so the token I get is valid from the first time it tries to connect, which is the first time I try to deploy. by the time I fixed the validation issues, its 40 minutes later and I've only got 20 minutes to monitor the deployment. – bytejunkie Sep 06 '16 at 10:47
  • I've just experienced the same - or very similar - issue causing mayhem in PowerShell and Visual Studio. Firstly, whatever I did in PowerShell it kept choosing the wrong subscription, even when I called Select-AzureRmSubscription with an explicit SubscriptionId(!) Then the same started happening in Visual Studio, before I got the same Expired Token error. The resolution for me was to call Clear-AzureRmContext with a -Scope of both Process and CurrentUser, before also calling Disable-AzureRmContextAutosave – AndyHerb Jul 05 '18 at 21:33
1

I bet it was a transient issue. I retried deployment (needed to modify my ARM template) and now it succeeded.

Please check your Azure Resource Group in the portal. You will likely have your resources up and running.

@Cleverguy25 provided an explanation of how I believe the deployment process work.

Igor Soloydenko
  • 11,067
  • 11
  • 47
  • 90
  • Seems it was transient, after deleting all the resources from my first run, I tried redeploying again with no changes to the ARM template, and it succeeded - it took about 45 minutes. – James Allen Mar 14 '16 at 20:24
  • Okay, so deploying the same ARM template the next day, I got `ExpiredAuthenticationToken` exception again, this time 11 minutes after kicking off the deploy. So it might not be a random one-off hiccup, rather a persistent and intermittent issue :( My guess is that VS periodically refreshes the authentication token in the background, maybe with an hour or two expiry time, and by random chance it may expire occasionally during a long running deployment? – James Allen Mar 15 '16 at 11:07
0

I am not sure, but I believe that the New-AzureRmResourceGroupDeployment uploads your template file and sets up a deployment to happen in the cloud. Then it queries the deployment to see if it is done and outputs the resources as they are created. Obviously those queries error when the token expires. But the deployment should continue.

You could ignore this error and query the deployment or resource group yourself, to see when it is done.

Cleverguy25
  • 493
  • 3
  • 14
  • If you are logging in via LiveId, there is no way to avoid interactive login. But if you login via Azure AD user, I think you can supply the password and login again. Also you can setup certificate based login with a ServicePrincipal as explained here: https://azure.microsoft.com/en-us/blog/azps-1-0/ – Cleverguy25 Mar 14 '16 at 18:57
  • Yes, it turns out the deployment did eventually succeed despite the error so I think you are right about the deployment continuing – James Allen Mar 14 '16 at 19:46
0

I follow this post, and simply execute 'Clear-AzureRmContext' this command, then reconnect to Azure, using 'connect-AzAccount', the issue resolved.

https://github.com/Azure/azure-powershell/issues/6585

Ju Chen
  • 11
  • 1
0

Open a new powershell and get the current metadata used to authenticate Azure Resource Manager requests using Clear-AzureRmContext.

This worked the magic for me.

Nkoro Joseph Ahamefula
  • 526
  • 3
  • 11