Net Core CosmosDb SDK - How to set the connection string after application startup?

Question

I want to connect to CosmosDb from an IOT Device (running as a Net Core 7 Web App) where the connection string is not known in advance of the application startup. In order to maintain some security and eliminate the need to store any connection strings in the app config, I'm looking at some options to pull this information down from Azure through IOT HUB.

In normal situations e.g. a standard webb app thats not running as an IOT Device, I would usually read the connection string from Azure Key Vault when initializing a singleton service for CosmosDb during the app startup, another approach is using Azure Managed Identity but this is only suitable for apps hosted in Azure i.e. not on premise.

I can't use either of the above in this scenario and looking to derive a solution in being able to present the connection string to the singleton service as and when it needs to fetch data from Cosmos, but this will always happen some time after the service was already initialized during the app startup.

Below is sample of the service when registered during app startup, I'm looking for some pointers in how best to modify this to accomodate the above requirements:

public class CosmosDbService : ICosmosDbService
{
    private static Database? Database;
    private readonly string _ohiAppSource;
    private readonly CosmosClient _cosmosClient;
    public readonly IConfiguration _configuration;
    public readonly ILogExtensionRepo _logExtensionRepo;

    public CosmosDbService(
        IConfiguration configuration,
        ILogExtensionRepo logExtensionRepo)
    {
        _configuration = configuration;
        _logExtensionRepo = logExtensionRepo;
        _ohiAppSource = _configuration["OhiAppSource"]!;

        _cosmosClient = new(
            // Fetching the connection string during service initialization is not an option for me here.
            connectionString: Environment.GetEnvironmentVariable("COSMOS_CONNECTION_STRING")!
        );

        CosmosClientOptions cosmosClientOptions = new()
        {
            MaxRetryAttemptsOnRateLimitedRequests = 3,
            MaxRetryWaitTimeOnRateLimitedRequests = TimeSpan.FromSeconds(60)
        };
    }

    private string? LogData { get; set; }

    protected Database GetDatabase(string databaseName)
    {
        var database = _cosmosClient.GetDatabase(databaseName);
        return database;
    }

    protected Container GetContainer(string databaseName, string containerName, string partitionKey)
    {
        Database = _cosmosClient.CreateDatabaseIfNotExistsAsync(databaseName).GetAwaiter().GetResult();

        if (String.IsNullOrEmpty(partitionKey))
        {
            partitionKey = "/id"; // Default partition key if not specified.
        }

        var container_createIfNotExists = Database.CreateContainerIfNotExistsAsync(containerName, partitionKey, 400).GetAwaiter().GetResult();
        return container_createIfNotExists;
    }
}

Interim Update

Some further research and more thinking...

The connection string for CosmosDb is not something that is expected to change too often, and by doing so we'd see a performance hit given we'd have to re-initialize the CosmosClient connection each time which always takes more time on the first request.

I think the prioroty solution must also factor in the service and it's availability when connection strings are rotated, which is inevitable at some point during the life cycle of the app run time, an IOT device in some scenarios could be running for extended periods of time, perhaps months.

I came across an interesting code sample from Azure which shows us how to implement a connection retry policy using Polly: https://github.com/Azure-Samples/reactive-keyrotation-handling

That code sample shows how to reinitialize the connection to Azure when the credentials are rotated, I've taken a few ideas from this.

Further reading on the web suggests singleton services are somewhat lazy in their operation i.e. service variables/paramaters only come into play the first time the service is actually used. If true my thoery is we could initialize the Cosmos service, but give it some duff values for the connection string during the app startup, then when I eventually call the service for the fist time, I just need to ensure that I have updated the variables with the correct values just in time for when the service needs to connect to Cosmos.

To manage key rotation, I would fetch the latest connection details through the device twin updated properties and forcefully reconnect to Cosmos. This would only be needed when the device twin properties have detected a change since the last update i.e. the keys had been rotated during a maintenance.

I'm will start testing the above and see what happens...