Azure VM Scaleset custom script extension not working - possibly failing to get VM identity?

Question

I'm attempting to deploy to my Virtual machine scale set using the custom script extension as below.

az vmss extension set --debug --name 'CustomScriptExtension' `
    --resource-group 'my-rg' `
    --publisher 'Microsoft.Compute'  `
    --version '1.9.5' `
    --vmss-name 'myvmss' `
    --settings '{\"commandToExecute\": \"powershell.exe ./download-package.ps1\", \"fileUris\": [\"https://[REDACTED].blob.core.windows.net/upload/download-package.ps1\"]}' `
    --protected-settings '{\"managedIdentity\": {\"objectId\": \"[REDACTED]\"}}'

When running I get the following error:

cli.azure.cli.core.azclierror : Deployment failed. Correlation ID: 73f4d16b-afe0-4373-8773-1d7dd7d26940. VM has reported a failure when processing extension 'CustomScriptExtension'. Error message: "Failed to download all specified files. Exiting. Error Message: Exception of type 'Microsoft.WindowsAzure.GuestAgent.Plugins.CustomScriptHandler.Downloader.MsiNotFoundException' was thrown."

More information on troubleshooting is available at https://aka.ms/VMExtensionCSEWindowsTroubleshoot Deployment failed. Correlation ID: 73f4d16b-afe0-4373-8773-1d7dd7d26940. VM has reported a failure when processing extension 'CustomScriptExtension'. Error message: "Failed to download all specified files. Exiting. Error Message: Exception of type 'Microsoft.WindowsAzure.GuestAgent.Plugins.CustomScriptHandler.Downloader.MsiNotFoundException' was thrown."

The file to be downloaded requires authentication so I have given the scale set a system assigned identity and granted it the Storage Blob Data Reader role on the storage account hosting the powershell file.

The custom extension logs on the VM suggest that it was unable to get the identity of the vm:

[7108+00000001] [11/20/2020 09:12:28.79] [INFO] Handler successfully enabled
[7108+00000001] [11/20/2020 09:12:28.80] [INFO] Loading configuration for sequence number 1
[7108+00000001] [11/20/2020 09:12:28.84] [INFO] HandlerSettings = ProtectedSettingsCertThumbprint: [REDACTED], ProtectedSettings: {[REDACTED]}, PublicSettings: {FileUris: [https://[REDACTED].blob.core.windows.net/upload/download-package.ps1], CommandToExecute: powershell.exe ./download-package.ps1}
[7108+00000001] [11/20/2020 09:12:29.26] [INFO] Downloading files specified in configuration...
[7108+00000001] [11/20/2020 09:12:30.90] [INFO] Attempting to get MSI from IMDS
[7108+00000001] [11/20/2020 09:12:31.04] [WARN] WebClient: non retryable error occurred System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
   at System.Net.WebClient.DownloadString(Uri address)
   at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClient.<>c__DisplayClass3_0.<DownloadStringWithRetries>b__0()
   at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClientWithRetryAbstract.ActionWithRetries(Action action)
[7108+00000001] [11/20/2020 09:12:31.14] [ERROR] Unknown exception occurred while attempting to get MSI token System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
   at System.Net.WebClient.DownloadString(Uri address)
   at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClient.<>c__DisplayClass3_0.<DownloadStringWithRetries>b__0()
   at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClientWithRetryAbstract.ActionWithRetries(Action action)
   at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.WebClient.DownloadStringWithRetries(Uri address)
   at Microsoft.WindowsAzure.GuestAgent.Plugins.MsiUtils.MsiProvider.GetMsiHelper(NameValueCollection queries)
[7108+00000001] [11/20/2020 09:12:31.14] [INFO] Msi was not obtained

I can retrieve the identity token from the metadata endpoint via Invoke-WebRequest -Method Get -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/' so that appears to be set up correctly.

Any advice on what the problem could be or how to further diagnose this issue would be greatly appreciated.

I have precisely the same problem but with UserAssigned Managed Identities. It appears thia is just plain broken as documented in this github issue: https://github.com/Azure/custom-script-extension-linux/issues/165 that has not been fixed over a year after being reported. — BrettRobi, Jun 17 '21 at 23:10

bhbrahma · Answer 1 · 2020-11-20T19:41:04.360

2

Here are the few fixes you can try

The object ID of the managed identity might be incorrect.
Please also move commandToExecute and FileUris into protected settings with managed identities.
If want to use system assigned managed identity, you don't need to pass a clientId or objectID, more info here https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-windows#property-managedidentity

edit: please explicitly pass an empty json object as settings when you add commandToExecute and fileUris to protected settings. Extensions would fail otherwise due to duplicated settings.

edited Nov 20 '20 at 19:41

answered Nov 20 '20 at 19:31

bhbrahma

21
2

Thanks for the response. Your points are correct and I have updated my code (moved commandToExecute and FileUris into protected settings and set managed identity to an empty block to use the system assigned identity). Unfortunatley it is still unable to access the file in blob storage, it's getting a 403 now. It works if I either make the blob public or use the access keys, so I don't think it's connectivity issue. I just can't get it to auth with managed identity. – Barker1889 Nov 24 '20 at 08:57

Azure VM Scaleset custom script extension not working - possibly failing to get VM identity?

1 Answers1