1

What is the recommended approach to automatically refresh credentials or application configuration in an Azure VM scale set?

Scenario

I have a horizontally-scalable application hosted in Azure that receives real-time streaming data. I've configured a simple CI/CD pipeline that builds a custom immutable image with the application binaries/libraries and then publishes it to an image gallery. From there, it's deployed to a scale set with a load balancer frontend to distribute traffic across the instances. At creation time, each instance fetches an application configuration file from blob storage and credentials from key vault.

Problem

When the application configuration or credentials are changed, those changes are not reflected in running instances until they are reimaged. As the application handles streaming data, I must manually effect a "rolling restart" by reimaging each instance one-by-one to ensure the application remains available while all instances are updated to the newest configuration.

Question

Azure provides a native "rolling upgrade" functionality to handle changes to scale set properties. I already use this when deploying new images, and it works very well. However, changes to the application configuration in blob storage or credentials in key vault obviously do not trigger a change to scale set properties. Is there a way to effect the same "rolling upgrade" process to account for external changes, without having to manually cycle through the instances and reimage them? Alternatively, is there a better approach to managing application configuration/credentials?

tfrederick74656
  • 229
  • 3
  • 12

1 Answers1

2

If you have scripts to fetch the application configuration file from blob storage and credentials from the key vault. It's recommended to uses the Custom Script Extension to run those scripts on Azure VM scale sets. The Custom Script Extension downloads and executes scripts on Azure VMs. This extension is useful for post-deployment configuration, software installation, or any other configuration/management task.

Nancy
  • 26,865
  • 3
  • 18
  • 34
  • Thanks for the information! Unfortunately Azure extensions also require the Azure Linux Agent (waagent) to be installed. I currently provision with pure cloud-init, and was hoping to avoid having to add the waagent package, as it tacks on about 100MB and a number of additional dependencies. I've tried to use cloud-init custom data in the same manner, but changes to the custom data definition don't trigger scale set updates. Are you aware of any other possible solutions? – tfrederick74656 Apr 13 '21 at 10:10
  • 2
    As cloud-init runs during the initial boot process, it does not trigger the scale set updates. In this case, you also could refer to [Install applications with OS update](https://learn.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-deploy-app#install-applications-with-os-updates) as the above links mentioned. – Nancy Apr 14 '21 at 01:44
  • 1
    Thanks for the assistance! I ended up deploying the Azure Linux Agent and using a simple custom script extension that clears the cloud-init data and reboots the VM. That allows me to toggle the [ForceUpdateTag](https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-windows#how-to-run-custom-script-more-than-once-with-cli) on the extension to trigger a rolling reboot/re-provision. – tfrederick74656 Apr 14 '21 at 19:29