Terraform provisioner cannot winrm to new built Windows VM on Azure

Question

I'm trying to provision a windows VM on Azure with Terraform and bootstrap it in the same time. The way I can think of is terraform provisioner. For testing purpose, I wrote the provisioner like this:

provisioner "remote-exec" {
    inline = [
      "md c:/terraform",
    ]
    connection {
      type     = "winrm"
      host   = "${azurerm_network_interface.vmstamp.private_ip_address}"
      user     = "${var.admin_username}"
      password = "${var.admin_password}"
      https    = false
      insecure = true
      timeout = "1m"
    }
}

The template successfully provisioned the VM, but yield an error when trying to connect the VM with WinRM.

azurerm_virtual_machine.vmstamp: Still creating... (5m50s elapsed)
azurerm_virtual_machine.vmstamp (remote-exec): Connecting to remote host via WinRM...
azurerm_virtual_machine.vmstamp (remote-exec):   Host: 10.237.249.146
azurerm_virtual_machine.vmstamp (remote-exec):   Port: 5985
azurerm_virtual_machine.vmstamp (remote-exec):   User: azadmin
azurerm_virtual_machine.vmstamp (remote-exec):   Password: true
azurerm_virtual_machine.vmstamp (remote-exec):   HTTPS: true
azurerm_virtual_machine.vmstamp (remote-exec):   Insecure: true
azurerm_virtual_machine.vmstamp (remote-exec):   NTLM: false
azurerm_virtual_machine.vmstamp (remote-exec):   CACert: false
azurerm_virtual_machine.vmstamp: Still creating... (6m0s elapsed)

Error: Error applying plan:

1 error(s) occurred:

* azurerm_virtual_machine.vmstamp: timeout - last error: unknown error Post https://10.237.249.146:5985/wsman: dial tcp 10.237.249.146:5985: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Several things I have tried:

1) Was thinking about firewall issue. But packer can launch connect to the new built windows VM on the same laptop with below code:

"communicator": "winrm",
"winrm_use_ssl": "true",
"winrm_insecure": "true",
"winrm_timeout": "3m",
"winrm_username": "packer",

2) Tried https=true and https=false, both failed.

3) Tried use_ntlm=true and use_ntlm=false, both failed.

4) Tried port=5985 and port=5986, both failed. 5986 is actually not listening in a new windows VM from market image.

what's the correct configuration for terraform provisioner?

Answer 1

I later figured out that winrm we need add a certificate to enable winrm. And to add the certificate, we need create a key store first and storage account as well. Packer does those steps for us, while terraform doesn't. So, we have to build those steps in terraform template to enable certificate and then enable winrm. The initiative was to bootstrap a windows vm without having to create additional resources like storage account or key store, because I need clean them up after provisioning which will complicate the template. If I have to create additional resource anyway, I would prefer using Azure VM extensions, because it doesn't need winrm port 5985 to be enabled on the NSGs, which is another big governance issue.

Answer 2

With your issue, I made the test. You can see the result for the windows image through Packer:

You can see that there is also a timeout for the winrm. And another test for Terraform I also got a timeout for the winrm. So I would suggest you can enable the winrm with a vm extension, make sure the port 5985 opened and then use the provisioner to do what you want.

Update

Maybe there is something missed so I did not succeed like you. But there is a process should pay attention to that Packer set the certificate's URL. The document here also shows that how to enable winrm with the certificate's URL if you want to use it. So I think Packer just do it himself, but Terraform does not do it for you.

Answer 3

@Charles Xu, Comment can not put too many charactors, I paste the output and code here. The packer script output:

azadmin@AZLAPSLOG1001:~/packer/winrmopen$ /usr/local/packer/packer build --var-file=variables.json Windows2016.json
azure-arm output will be in this color.

==> azure-arm: Running builder ...
    azure-arm: Creating Azure Resource Manager (ARM) client ...
    azure-arm: You have provided Object_ID which is no longer needed, azure packer builder determines this dynamically from the authentication token
==> azure-arm: Using existing resource group ...
==> azure-arm:  -> ResourceGroupName : '###Resource Group Replacement####'
==> azure-arm:  -> Location          : 'canadacentral'
==> azure-arm: Validating deployment template ...
==> azure-arm:  -> ResourceGroupName : '###Resource Group Replacement####'
==> azure-arm:  -> DeploymentName    : 'pkrdp5u8k15bqf0'
==> azure-arm: Deploying deployment template ...
==> azure-arm:  -> ResourceGroupName : '###Resource Group Replacement####'
==> azure-arm:  -> DeploymentName    : 'kvpkrdp5u8k15bqf0'
==> azure-arm: Getting the certificate's URL ...
==> azure-arm:  -> Key Vault Name        : 'pkrkv5u8k15bqf0'
==> azure-arm:  -> Key Vault Secret Name : 'packerKeyVaultSecret'
==> azure-arm:  -> Certificate URL       : 'https://pkrkv5u8k15bqf0.vault.azure.net/secrets/packerKeyVaultSecret/d1a2c8fe28c34b73b65bb53b7a9ea34f'
==> azure-arm: Setting the certificate's URL ...
==> azure-arm: Validating deployment template ...
==> azure-arm:  -> ResourceGroupName : '###Resource Group Replacement####'
==> azure-arm:  -> DeploymentName    : 'pkrdp5u8k15bqf0'
==> azure-arm: Deploying deployment template ...
==> azure-arm:  -> ResourceGroupName : '###Resource Group Replacement####'
==> azure-arm:  -> DeploymentName    : 'pkrdp5u8k15bqf0'
==> azure-arm: Getting the VM's IP address ...
==> azure-arm:  -> ResourceGroupName   : '###Resource Group Replacement####'
==> azure-arm:  -> PublicIPAddressName : 'pkrip5u8k15bqf0'
==> azure-arm:  -> NicName             : 'pkrni5u8k15bqf0'
==> azure-arm:  -> Network Connection  : 'PrivateEndpoint'
==> azure-arm:  -> IP Address          : '10.236.8.92'
==> azure-arm: Waiting for WinRM to become available...
    azure-arm: #< CLIXML
    azure-arm: WinRM connected.
    azure-arm: <Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><Obj S="progress" RefId="0"><TN RefId="0"><T>System.Management.Automation.PSCustomObject</T><T>System.Object</T></TN><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj><Obj S="progress" RefId="1"><TNRef RefId="0" /><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj></Objs>
==> azure-arm: Connected to WinRM!
==> azure-arm: Provisioning with Powershell...

You can see very clear that it successfully connected to the new built VM via winrm. Below are the script I used for packer: Variables

{
  "tenant_id": "###Tenant ID Replacement ###",
  "subscription_id": "###Subscription ID Replacement",
  "region_name": "canadacentral",
  "virtual_network_rg": "###VNET ResourceGroupName replacement ####",
  "virtual_network_name": "###VNET Name Replacement ####",
  "virtual_network_subnet_name": "Presentation",
  "build_resource_group_name": "###Resource Group Replacement####",
  "image_resource_group_Name": "###Resource Group Replacement####",
  "chef_server": "https://chefserver01....",
  "object_id": "f3b76eb8-87e6-42d7-9d12-dafa5b124d90",
  "chef_org_url": "https://artifactory....",
  "chef_org_name": "org name replacement",
  "chef_client_binary": "https://artifactory.......",
  "chef_first_boot": "https://artifactory.......",
  "chef_client_config": "https://artifactory.......",
  "wsus_server": "http://WSUS....",
  "wsus_script": "https://artifactory......",
  "wsus_task": "https://artifactory.....",
  "chef_client_bootstrap": "https://artifactory.....",
  "chef_checkin_task": "https://artifactory......",
  "chef_checkin_xml": "https://artifactory......",
  "image_build_number": "1811261812",
  "client_id": "###Client ID replacement",
  "client_secret": "###Client Secret replacement"
}

windows.json

{
  "variables": {
    "client_secret": "{{user `client_secret`}}"
  },  
  "builders": [{
  "type": "azure-arm",
  "client_id": "{{user `client_id`}}",
  "client_secret": "{{user `client_secret`}}",
  "tenant_id": "{{user `tenant_id`}}",
  "subscription_id": "{{user `subscription_id`}}",
  "object_id": "{{user `object_id`}}",
  "build_resource_group_name": "{{user `build_resource_group_name`}}",

  "managed_image_resource_group_name": "{{user `image_resource_group_Name`}}",
  "managed_image_name": "Windows2016-dev-{{user `image_build_number`}}",

  "os_type": "Windows",
  "image_publisher": "MicrosoftWindowsServer",
  "image_offer": "WindowsServer",
  "image_sku": "2016-Datacenter",
  "image_version":"latest",
  "os_disk_size_gb":"128", 
  "vm_size": "Standard_E4S_v3",
  "virtual_network_resource_group_name":"{{user `virtual_network_rg`}}",
  "virtual_network_name":"{{user `virtual_network_name`}}",
  "virtual_network_subnet_name":"{{user `virtual_network_subnet_name`}}",



  "communicator": "winrm",
  "winrm_use_ssl": "true",
  "winrm_insecure": "true",
  "winrm_timeout": "3m",
  "winrm_username": "packer",

  "azure_tags": {
      "dept": "Platform Engineering",
      "task": "Daily Image Build",
      "CostCenter": "3357",
      "UseCase": "Standard Image Builds - GSPE",
      "ReleaseDate": "{{isotime `2006-01-02`}}"
  }
  }],
  "provisioners": [ 
    {
    "type": "powershell",
    "environment_vars": [
      "chef_org_url={{user `chef_org_url`}}",
      "chef_org_name={{user `chef_org_name`}}",
      "chef_client_binary={{user `chef_client_binary`}}",
      "chef_client_config={{user `chef_client_config`}}",
      "chef_first_boot={{user `chef_first_boot`}}",
      "chef_client_bootstrap={{user `chef_client_bootstrap`}}",
      "wsus_server={{user `wsus_server`}}",
      "wsus_script={{user `wsus_script`}}",
      "wsus_task={{user `wsus_task`}}",
      "chef_setrunonce={{user `chef_setrunonce`}}",
      "chef_server={{user `chef_server`}}",
      "chef_checkin_task={{user `chef_checkin_task`}}",
      "chef_checkin_xml={{user `chef_checkin_xml`}}"

    ],  
    "inline": [
      "$chef_org_client=$ENV:chef_org_name + '-validator'",
      "$chef_org_validator_filename=$chef_org_client + '.pem'",
      "$chef_org_validator_local='c:\\chef\\' + $chef_org_validator_filename",
      "$chef_org_validator_url=$ENV:chef_org_url + $chef_org_validator_filename",


      "Write-Host Setting TLS 1.2",
      "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12",

      "Write-Host Download Chef Client, Org Validator and First Boot",
      "Write-Host Full ORG Validator PEM FILE - $chef_org_validator_url",
      "Write-Host Chef Org Name - $env:chef_org_name",
      "Write-Host WSUS Server Name - $env:wsus_server",
      "Write-Host Chef Server - $env:chef_server",
      "md $env:windir\\Temp\\packer-chef-client -Force",
      "md $env:SystemDrive\\chef\\ -Force",
      "iwr -Uri $chef_org_validator_url -Outfile $chef_org_validator_local",
      "iwr -Uri $ENV:chef_client_binary -Outfile $env:windir\\Temp\\packer-chef-client\\chef.msi",
      "iwr -Uri $ENV:chef_client_config -Outfile $env:SystemDrive\\chef\\client.rb",
      "iwr -Uri $ENV:chef_first_boot -Outfile $env:SystemDrive\\chef\\first-boot.json",
      "iwr -Uri $ENV:chef_client_bootstrap -Outfile $env:windir\\Temp\\packer-chef-client\\chef_client_bootstrap.ps1",
      "iwr -Uri $ENV:chef_checkin_task -Outfile $env:windir\\Temp\\packer-chef-client\\chef_checkin_task.ps1",
      "iwr -Uri $ENV:chef_checkin_xml -Outfile $env:windir\\Temp\\packer-chef-client\\ChefRunOnce.xml",
      "iwr -Uri $ENV:wsus_script -Outfile $env:windir\\Temp\\packer-chef-client\\WSUSUpdate.ps1",
      "iwr -Uri $ENV:wsus_task -Outfile $env:windir\\Temp\\packer-chef-client\\DownloadStageChef-modded.ps1",

      "Write-Host Install Chef Client",
      "Start-Process 'msiexec' -ArgumentList '/qb /i C:\\Windows\\Temp\\packer-chef-client\\chef.msi' -NoNewWindow -Wait",

      "Write-Host Set Regional WSUS Server",
      "reg add 'HKEY_LOCAL_MACHINE\\SOFTWARE\\Policies\\Microsoft\\Windows\\WindowsUpdate' /v WUServer /t REG_SZ /d $ENV:wsus_server /f",
      "reg add 'HKEY_LOCAL_MACHINE\\SOFTWARE\\Policies\\Microsoft\\Windows\\WindowsUpdate' /v WUStatusServer /t REG_SZ /d $ENV:wsus_server /f",

      "Write-Host Run WSUS Update",
      "C:\\Windows\\Temp\\packer-chef-client\\DownloadStageChef-modded.ps1",

      "Write-Host Chef RunOnce Task",
      "C:\\Windows\\Temp\\packer-chef-client\\chef_checkin_task.ps1 -chef_org_name $env:chef_org_name -chef_server $env:chef_server"
    ]
  },
  {
    "type": "powershell",
    "inline": [
      "if( Test-Path $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml ){ rm $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml -Force}",
      "& $env:SystemRoot\\System32\\Sysprep\\Sysprep.exe /oobe /generalize /quiet /quit",
      "while($true) { $imageState = Get-ItemProperty HKLM:\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup\\State | Select ImageState; if($imageState.ImageState -ne 'IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE') { Write-Output $imageState.ImageState; Start-Sleep -s 10  } else { break } }"
    ]
  }
]
}