4

Background

Our applications are in .Net core. We use Service Fabric for cluster management. We have multiple VM scale sets. We use MSI during Application startup. The code has been working for more than 6 months. We are using code as outlines in this link to get the access token.

Issue:

Recently we added Azure KVVM extension to our VM scale sets. This was done to get auto renewed cluster certificates from the KeyVault. After we made this change our applications are unable to get Access token.

We added KVVM extension in our service fabric ARM deployment template as suggested by Service Fabric team. The deployment went Through and we saw that our VMSS were in good state.

KVVm settings in the VMSS looks like this - Version 0.2

{
  "secretsManagementSettings": {
    "pollingIntervalInS": "86400",
    "observedCertificates": [
      "https://azsc-eu-padev1.vault.azure.net/secrets/clusterCertificate"
    ],
    "requireInitialSync": true
  }
}

The errors that we are getting for getting access Token on service start up is this - 

Failed to get configuration key vault access token. Request to http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=2eabacbd-4fb2-4d7c-b3ab-b4e3fc7a53e7&resource=https://vault.azure.net failed with BadRequest: StatusCode: 400, ReasonPhrase: 'Bad Request', Version: 1.1, Content: System.Net.Http.HttpConnection+HttpConnectionResponseContent, Headers:
{
Server: Microsoft-IIS/10.0
Date: Thu, 31 Jan 2019 01:24:03 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 68
}

The VMSS is showing error 

Provisioning of VM extension 'KVVMExtension' has timed out. Extension installation may be taking too long, or extension status could not be obtained.

Any help on how to get KVVM extension and MSI work properly will be great.

Thanks a bunch.

Scooby
  • 635
  • 3
  • 9
  • 21

1 Answers1

2

Azure resources may have system- or user-assigned managed identities. VMSS supports both types, as well as mixing them. IMDS (the metadata endpoint shown above: http://169.254.169.254/metadata/identity/oauth2/token?...) accepts specifying a AAD client id representing the identity the caller wants to assume/impersonate. If note provided, the behavior/response from IMDS depends on the types and number of managed identities associated with the VMSS.

Earlier versions of the KV VM extension only supported the 'implicit' model - a client id could not be specified in the extension's settings, and so the outbound call to IMDS fails.

Since you specified the 'requireInitialSync' flag, the KVVM extension will block on its first run - executed during VMSS start-up - until it's able to retrieve the specified secret(s). The failure to obtain a token causes the failure to retrieve a secret, which blocks and causes the VM agent to wait indefinitely on installing the KVVM extension … and so on.

You mentioned Service Fabric - the end-to-end steps to enable certificate rollover for Azure Service Fabric clusters is as follows (high-level first, details steps after): - use a user-assigned identity - pre-ACL the identity to the vault - declare the KVVM extension with 'requireInitialSync' - declare the SFVM extension with 'provisionAfter' the KVVM ext

Relevant template excerpts: - declare the UAI (under 'resources'):

`{
  "type": "Microsoft.ManagedIdentity/userAssignedIdentities",
  "name": "[parameters('userAssignedIdentityName')]",
  "apiVersion": "2018-11-30",
  "location": "[resourceGroup().location]"
},`
  • grant the UAI access to the vault:

{ "type": "Microsoft.KeyVault/vaults/accessPolicies", "name": "[concat(parameters('keyVaultName'), '/add')]", "apiVersion": "2018-02-14", "properties": { "accessPolicies": [ { "tenantId": "[reference(variables('userAssignedIdentityResourceId'), '2018-11-30').tenantId]", "objectId": "[reference(variables('userAssignedIdentityResourceId'), '2018-11-30').principalId]", "dependsOn": [ "[variables('userAssignedIdentityResourceId')]" ], "permissions": { "secrets": [ "get", "list" ] } } ] } },

  • assign the UAI to the VMSS:

    "identity": { "type": "UserAssigned", "userAssignedIdentities": { "[variables('userAssignedIdentityResourceId')]": {} } },

  • declare the KV and SF VM extensions, respectively (under "virtualMachineProfile"\ "extensionProfile"\"extensions"):

{ "name": "KVVMExtension", "properties": { "publisher": "Microsoft.Azure.KeyVault", "type": "KeyVaultForWindows", "typeHandlerVersion": "1.0", "autoUpgradeMinorVersion": true, "settings": { "secretsManagementSettings": { "pollingIntervalInS": "[parameters('kvvmextPollingInterval')]", "linkOnRenewal": false, "observedCertificates": "[parameters('kvvmextObservedCertificates')]", "requireInitialSync": true } } } },

and, for SF (abbreviated):

{ "name": "[concat('ServiceFabricNodeVmExt','_vmNodeType0Name')]", "properties": { "type": "ServiceFabricNode", "autoUpgradeMinorVersion": true, "provisionAfterExtensions" : [ "KVVMExtension" ], "settings": { "certificate": { "commonNames": [ "[parameters('certificateCommonName')]" ], "x509StoreName": "[parameters('certificateStoreValue')]" } }, "typeHandlerVersion": "1.0" }

Note the 'provisionAfter' property, as well as the certificate declaration (by subject Common Name.)

dragosav
  • 17
  • 3