Azure VM custom script extension - failing randomly

Question

I have been troubleshooting the CustomScriptExtension using Terraform I am using it to deploy and run a powershell script that works 80% of the times I deploy. I have tested 48 VM deployments. When it fails on say VM2 for example the next time I run it, it is successful on VM2

I am using Azure DevOps to deploy it.

My thoughts:

• In my opinion we can take Terraform out of the picture as the issue seems to be at the Azure level

Here is the code in my Terraform Module:

resource "azurerm_virtual_machine_extension" "extension" {
  name                 = "Proxy-Settings"
  location             = var.location
  resource_group_name  = var.resource_group_name
  virtual_machine_name = azurerm_virtual_machine.windows_vm.name
  publisher            = "Microsoft.Compute"
  type                 = "CustomScriptExtension"
  type_handler_version = "1.9"

  settings = <<SETTINGS
    {
        "fileUris": ["https://this_is_a_secret.blob.core.windows.net/scripts/autoProxyConfig.ps1"]
    }
    SETTINGS

  protected_settings = <<PROTECTED_SETTINGS
    {
      "commandToExecute"  : "powershell -ExecutionPolicy Unrestricted -File autoProxyConfig.ps1",
      "storageAccountName": "this_is_a_secret",
      "storageAccountKey" : "this_is_a_secret"
    }
    PROTECTED_SETTINGS
}

Note that the error message from the DevOps pipeline is pretty useless:

2020/03/22 19:31:36 [ERROR] module.the_vm_2: eval: *terraform.EvalApplyPost, err: Code="VMExtensionProvisioningError" Message="VM has reported a failure when processing extension 'Proxy-Settings'. Error message: \"Finished executing command\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSEWindowsTroubleshoot

And here is another version of this error:

2020-03-22T19:31:37.5021338Z [0m on .terraform/modules/the_vm_2/main.tf line 79, in resource "azurerm_virtual_machine_extension" "extension": 2020-03-22T19:31:37.5022143Z 79: resource "azurerm_virtual_machine_extension" "extension" [4m{[0m 2020-03-22T19:31:37.5022853Z [0m 2020-03-22T19:31:37.5023130Z [0m[0m 2020-03-22T19:31:37.5023249Z 2020-03-22T19:31:37.5060259Z ##[error]Bash exited with code '1'.

Here is the PowerShell script:

    #Ensure network adapter is not set to Public
$networkAdapter = Get-NetAdapter | Where-Object {$_.Status -match "Up"} | Get-NetIPAddress | Where-Object {$_.AddressFamily -match "IPv4" -AND $_.PrefixOrigin -match "Dhcp"}
$networkCategory = $networkAdapter | Get-NetConnectionProfile | Select-Object -ExpandProperty NetworkCategory
if ($networkCategory -eq "Public") {
    $interfaceIndex = $networkAdapter | Select-Object -ExpandProperty InterfaceIndex
    try {
        Write-Output "Changing connection profile for network adapter to private as it is not currently configured as DomainAuthenticated or Private..."
        Set-NetConnectionProfile -InterfaceIndex $interfaceIndex -NetworkCategory Private
    }
    catch {
        Write-Error -Exception "$(Get-TimeStamp): What didn't happen" -Message "$_.Exception.Message"
    }    
}

#Prepare pacURL and hexValue parameters
$hexOutput = ""
#Get the current IP Address of the machine, filtering out irrelevant IP addresses
$IPAddr = $networkAdapter | Select-Object -ExpandProperty IPAddress
$splitIP = $IPAddr.split('.')

if ($splitIP[1] -eq "185") {
    $secondOctet = "184"
}
else {
    $secondOctet = $splitIP[1] 
}

$subnetOctets = @($splitIP[0],$secondOctet)
foreach ($octet in $subnetOctets) {
    $octet | Format-Hex | Select-Object -ExpandProperty Bytes | foreach-object {$hexOutput+= [System.Convert]::ToString($_,16)}
    $hexOutput += "2e"
}
$pacURL = "http://"+$splitIP[0]+"."+$secondOctet+".228.4/proxySettings.pac"
$hexValue = "46000000090000000d000000000000000000000025000000687474703a2f2f"+$hexOutput+"3232382e342f70726f787953657474696e67732e7061630000000000000000000000000000000000000000000000000000000000000000"

try {
    Write-Output "Setting proxy settings for machine..."
    #Add these registry values
    &REG ADD "HKLM\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings" /v ProxySettingsPerUser /t REG_DWORD /d "0" /f
    &REG ADD "HKLM\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings" /v EnableAutoProxyResultCache /t REG_DWORD /d "0" /f
    &REG ADD "HKLM\Software\Microsoft\Windows\CurrentVersion\Internet Settings" /v ProxyEnable /t REG_DWORD /d "1" /f
    &REG ADD "HKLM\Software\Policies\Microsoft\Internet Explorer\Control Panel" /v Autoconfig /t REG_DWORD /d "1" /f
    &REG ADD "HKLM\Software\Policies\Microsoft\Internet Explorer\Control Panel" /v Proxy /t REG_DWORD /d "1" /f
    &REG ADD "HKLM\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections" /v DefaultConnectionSettings /t REG_BINARY /d $hexValue /f
    &REG ADD "HKLM\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections" /v SavedLegacySettings /t REG_BINARY /d $hexValue /f
    &REG ADD "HKLM\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\Internet Settings\Connections" /v DefaultConnectionSettings /t REG_BINARY /d $hexValue /f
    &REG ADD "HKLM\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\Internet Settings\Connections" /v SavedLegacySettings /t REG_BINARY /d $hexValue /f
    &REG ADD "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings" /v AutoConfigURL /t REG_SZ /d $pacURL /f

    #Delete these registry values if they exist
    &REG DELETE "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections" /v WinHttPSettings /f
    &REG DELETE "HKLM\Software\Microsoft\Windows\CurrentVersion\Internet Settings" /v ProxyServer /f
    &REG DELETE "HKLM\Software\Microsoft\Windows\CurrentVersion\Internet Settings" /v ProxyOverride /f
}
catch {
    Write-Error -Exception "Unable to apply proxy registry settings!" -Message "$_.Exception.Message"
}

exit 0