Cannot RDP to an instance of virtual machine scale set of a Service Fabric cluster in Azure

Question

I have created a Service Fabric cluster by using the well-known azure-quickstart-templates/service-fabric-secure-cluster-5-node-1-nodetype template.

During the deployment I have specified the adminUsername and adminPassword strings as the ARM template parameters. I have just used 2 long strings of random characters and written them down for later.

The deployment has succeeded and I can for example open the SF cluster explorer webpage with the famous green circles.

When I try to connect to one of the nodes of the VMSS though, in order to check my SF application logs, the RDP connection is being refused:

I use the Loader public IP address for the RDP connection and have verified that there is an LB rule for that:

When I enter the adminUsername and adminPassword strings I had specified during the ARM template deployment, the RDP connection prepends the domain of my corporate notebook.

So I have tried prepending the LB public ip address followed by a backslash in front of the adminUsername, but that does not work either.

Also I have tried .\adminUsername (takes my corporates notebook's domain which is wrong) and \adminUsername and nt1vm_1\adminUsername (still does not connect):

How can I please RDP-connect to the VMSS instances of my SF cluster?

I have also created the Github issue #7684 for my problem.

UPDATE:

Cannot connect with Microsoft Remote Desktop for mac OS either:

UPDATE 2:

Apologies for oversimplifying my problem description -

I have actually not written the RDP credentials to paper, but generated and saved them in a Key Vault, then outputted them as the pipeline variables (I know, not so secure...) and passed them to the SF template:

So this is not the issue of me writing down wrong credentials. And this YAML file is used by 5 pipelines, to fill 5 different RGs (with CosmosDb, Key Vault, SF) - all having the above RDP connection problem.

UPDATE 3:

Because I have 3 nodes in the SF cluster, I have tried using nt1vm_0\username, nt1vm_1\username, nt1vm_2\username and even nt1vm\username as username at the RDP dialog, but alas that does not work:

"I have just used 2 long strings of random characters and written them down for later" - is this issue reproducible? Or is there a chance that the credentials could be incorrect? — Oliver, Jun 15 '20 at 18:24
Actually, in the Azure pipeline before deploying the SF cluster, I have [deployed a Key Vault, with 2 secrets](https://github.com/Azure/azure-quickstart-templates/tree/master/201-key-vault-secret-create) generated by `uniqueString()` plus some capital letters and digits to satisfy the password complexity. And I have passed them as pipeline vars for the SF template. So this is not the issue and apologies for simplifying my description. I did not write them down on paper, but instead saved as 2 secrets in the KV before deploying SF. — Alexander Farber, Jun 15 '20 at 18:54
Concerning the statement "outputted them as the pipeline variables ... and passed them to the SF template" -- does that mean you are using Azure DevOps release pipelines to deploy? The reason I ask is because I got burned by using PowerShell in a pipeline and consumed a variable with a password using `"$(sfcPassword)"`. The resolved variable contained a '$' within the value which was then treated as a PowerShell variable within the password string. — Stringfellow, Jun 15 '20 at 22:45
did you try `nt1vm_0\adminUsername` on port 3389? (you said `nt1vm_1\adminUsername` ) — LoekD, Jun 16 '20 at 07:09
Thanks for replying (upvoted), but this does not work - please see the Update 3 — Alexander Farber, Jun 16 '20 at 08:05
did you try doing a [password resethttps://learn.microsoft.com/en-us/azure/service-fabric/scripts/service-fabric-powershell-change-rdp-user-and-pw]() on the scaleset to see if the credentials may have been changed somehow? — LoekD, Jun 16 '20 at 14:12
@LoekD this is a good workaround too! Please submit as answer and I will accept :-) — Alexander Farber, Jun 18 '20 at 09:23

score 1 · Accepted Answer · edited Jun 19 '20 at 09:21

Try doing a password reset on the scaleset to see if the credentials may have been changed somehow.

Login-AzAccount
Get-AzSubscription
Set-AzContext -SubscriptionId 'yourSubscriptionID'

$nodeTypeName = 'nt1vm'
$resourceGroup = 'sfclustertutorialgroup'
$publicConfig = @{'UserName' = 'newuser'}
$privateConfig = @{'Password' = 'PasSwo0rd$#!'}
$extName = 'VMAccessAgent'
$publisher = 'Microsoft.Compute'
$node = Get-AzVmss -ResourceGroupName $resourceGroup -VMScaleSetName $nodeTypeName
$node = Add-AzVmssExtension -VirtualMachineScaleSet $node -Name $extName -Publisher $publisher -Setting $publicConfig -ProtectedSetting $privateConfig -Type $extName -TypeHandlerVersion '2.0' -AutoUpgradeMinorVersion $true

Update-AzVmss -ResourceGroupName $resourceGroup -Name $nodeTypeName -VirtualMachineScaleSet $node

Alexander Farber · Answer 2 · 2020-06-18T08:13:53.250

Here is my own answer, maybe it will help someone too -

It seems to be a bug in SF Cluster (we use Service Fabric version 7.1.417.9590) ARM template deployment.

We have noticed, that RDP worked if you "Reimage" all instances in the VMSS:

Without the reimaging the RDP connection was refused as if the credentials were wrong.

In our Azure pipeline the RDP credentials were generated by the following code in the Key Vault ARM template (the capital letters and digits to satisfy password complexity requirements + a unique string based on deployment name):

"variables": {
    "RdpUsername": "[concat('ABC123', uniqueString(deployment().name, 'RdpUsername'))]",
    "RdpPassword": "[concat('ABC123', uniqueString(deployment().name, 'RdpPassword'))]"
},
"outputs": {
    "RdpUsername": {
        "type": "string",
        "value": "[variables('RdpUsername')]"
    },
    "RdpPassword": {
        "type": "string",
        "value": "[variables('RdpPassword')]"
    },
    "keyvaultId": {
        "type": "string",
        "value": "[resourceId('Microsoft.KeyVault/vaults', variables('keyvaultName'))]"
    }
},

The uniqueString is based on deployment name and thus is changed with each pipeline deployment (that is why there are different secret versions in the screenshot below):

Then the above secrets where passed to the pipeline task deploying SF cluster:

- task: AzureResourceManagerTemplateDeployment@3
  displayName: 'Deploy SF cluster'
  inputs:
    deploymentScope: 'Resource Group'
    subscriptionId: '${{ parameters.SubscriptionId }}'
    azureResourceManagerConnection: '${{ parameters.ArmConnection }}'
    action: 'Create Or Update Resource Group'
    resourceGroupName: '${{ parameters.ResourceGroupName }}'
    location: '${{ parameters.ResourceLocation }}'
    templateLocation: 'Linked artifact'
    csmFile: '$(Build.SourcesDirectory)/pipelines/templates/sfcluster.json'
    csmParametersFile: '$(Build.SourcesDirectory)/pipelines/templates/sfcluster-params.json'
    overrideParameters: '-rdpUsername $(RdpUsername) -rdpPassword $(RdpPassword) -certificateThumbprint $(Thumbprint) -sourceVaultResourceId $(KeyvaultId) -certificateUrlValue $(SecretId)' 
    deploymentMode: 'Incremental'

Because there seems to be a bug, the RDP connection still required the older pair of the RDP credentials.

So our workaround is to change the RDP credentials to a more stable string, based on the RG name:

"variables": {
    "RdpUsername": "[concat('Ccg1', uniqueString(resourceGroup().name, 'RdpUsername'))]",
    "RdpPassword": "[concat('Ccg2', uniqueString(resourceGroup().name, 'RdpPassword'))]"
},

And now RDP connections work (with username nt1vm_0\RdpUsername etc.):

Cannot RDP to an instance of virtual machine scale set of a Service Fabric cluster in Azure

2 Answers2