How to load configuration settings again after service starts

Question

My application is hosted in AKS, the secrets are stored in Azure secrets storage. What my client has is created a volume which stores /mnt/secrets-store that directory contains all of the secrets for my application.

For example:

• RabbitMq--Password
• DB--ConnectionString

Apparently this is the standard practice with my current client. There solution is then to read the file every time they want to load the settings. This is all set up in the healm charts.

   volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
        {{- end}}

A number of my services are loaded as singletons at startup which means the settings need to be available at startup.

Note: My service has no direct access to Azure key vault and this is not something that can be changed. All I have access to are these files.

What I did was create a custom IConfigurationBuilder called UseAzureKeyVaultMountedSecrets everything is very strait forward with this. It just checks for the folder and then loads the secrets in replacing whatever is in appsettings..json for that key being the name.

  public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .UseAzureKeyVaultMountedSecrets()
                .UseSerilog((context, config) => config.ReadFrom.Configuration((context.Configuration)))
                .ConfigureWebHostDefaults(webBuilder => { webBuilder.UseStartup<Startup>(); });

Locally to test I just created the folder with the files in it and it works like a charm. However when hosted on AKS I started having issues. My health checks state that it can detect the mnt dir. But its not reading them.

After quite a bit of debugging, I found that the issue is that the mnt dir is not available when the microservice is starting up in the pod. Its first after the pod is running that it can access the mnt dir.

Is there a solution for this?

• How can I get it to mnt the volume before start up in AKS?
• Is it possible to load the settings again after the service has started? If it is. then I'm wondering if I could then create something like an IHostedService which would just keep trying to access it and then stop once it has grabbed them.

The main part of the problem is that these settings are used to configure all of the singletons running in the system. If I wait until everything has loaded, they have already been configured.

Answer 1

@christian-sauer is correct I believe your "problem" is that it takes some time for these "virtual" files to show up as they are mapped from Azure Vault. ie the mount is complete and so pod initialization moves on, but the files take time to show up while the secrets are being projected.

So you need to make your app wait until they show up. I'm not sure what you mean by "The pod wont finish loading". It will be fine. It's just that your process will be doing a little extra work while it starts up.

So, just add a little loop that checks for there being files in that folder, then continue. It'll take a little effort to make it robust and you will need to experiment and observe how these files appear. ie do they all appear at once? Or do they appear over a period of time? Maybe you count the files, wait, count again. If the count is stable for some period of time (500ms?) then break the loop and continue. If it takes too long (10s, 30s?) then throw and the process (and pod) will fail. Which is the correct behavior.

You are right to be thinking about health checks, but... config is a tricky thing. As you have pointed out it is required to be available before other things can be bootstrapped.

Maybe use the IHostBuilder ConfigureHostConfiguration extension to do you file check loop. This is early enough such that startup is blocked before anything refers to config. (Note: there are two phased: Host config; App config).

You are right, this does block the process while the check is working. This is completely fine.

---

For completeness... you might think of using IOptionsMonitor instead (updates IConfiguration when the underlying provider detects a change. ie file update). Seems "good", but...

1. will it be updated by the time your other pieces need it? You might be able to refactor these pieces to hook into the Changed event and re-init themselves. Here be dragons.
2. In k8s the "files" you're reading are often symlinks. The SystemFileWatcher will not catch changes as the symlink attrs do not change, only the data being pointed to.

So, KISS, just block the process until the files show up