Passing KeyVault secrets to .net core 2 xUnit/MsTest in VSTS

Question

I have several secrets stored in Azure KeyVault. Unfortunately I cannot find a way to pass parameters to my .net Core 2.0 test run via VSTS (Visual Studio Team Services)

Documentation says that Keyvault secrets can only be supplied via VSTS variables - fair enough - but how do I actually do this? All the information I can find on web seems outdated or doesn't work.

For example - consider RunSettings file:

<RunSettings>
    <TestRunParameters>  
        <Parameter name="webAppUrl" value="http://localhost" /> 
        <Parameter name="webAppUserName" />
        <Parameter name="webAppPassword" />
  </TestRunParameters> 
</RunSettings>

I tried passing values for last 2 parameters via cmd line as follow:

vsts.console MyTest.dll /Settings:vsts.runsettings -- -webAppUserName foo
vsts.console MyTest.dll /Settings:vsts.runsettings -- webAppUserName=foo
dotnet test -s vsts.runsettings -- -webAppUserName foo
dotnet test -s vsts.runsettings -- webAppUserName=foo

but this has no effect - the webAppUserName value remains null (I can see a value for webAppUrl, so I know my code is right!)

I've also tried both the VSTS "dotnet test" task as well as the "VsTest" task from my VSTS 2017 build. The VsTest provides a Override test run parameters setting - and as per the tooltip, I tried:

-webAppUserName user -webAppPassword somethingSecret

Again - no effect!

Originally I used xUnit and had exactly the same issue - i.e. coudn't figure out a way to pass parameters via VSTS - so I tried MSTest, but same issue.

Going back to my original issue of injecting KeyVault secrets Locally via VS Studio I was able to just do the following from my test:

var config = new ConfigurationBuilder()
                .AddJsonFile("appsettings.json", true, true)
                ...
                .AddAzureKeyVault(...)

Unfortunately, when run via VSTS, the test runner just hangs (i.e. I had to stop it after several minutes) with no log output for test step if I use either

.AddAzureKeyVault(...) or .AddEnvironmentVariables()

So I tried using VSTS Variable Groups - and linking that to KeyVault. However, keyvault secrets are not accessible via environment variables (using Enviroment.GetEnvironmentVariable(...) directly from C#) - so that's no good. They say you can only pass these to tasks via VSTS variables...hence my problem!

Aside: Even if I could use environment variables, it's not optimal because when using .AddAzureKeyVault() I can supply a custom IKeyVaultSecretManager to, for example, replace a special delimiter with the ':' character - this means that I can nest my json config values - e.g. if I had this in my config file:

{ "A" : { "B" : "somevalue" } }

then using normal configuration builder I can access the above via config["A:B"]. Unfortunately KeyVault doesn't like the ":" character - so you have to replace it with something like "--" and then use a custom IKeyVaultSecretManager to replace "--" with ":" (which works great and ensures that variables are properly overridden based on the order of providers registered in config builder)

Please help! All I wanted for Christmas was not to put my KeyVault secrets into Git... but the VSTS KeyVault grinch is spoiling my fun... surely I'm missing something??

Find the repository for `AddAzureKeyVault` on Github and review its source to see what it does. Maybe it provides some hints about what you need to make a workaround. This smells like a design issue with tight coupling to dependencies that are difficult to mock/stub/fake for testing. — Nkosi, Jan 04 '18 at 11:11
@nkosi Thanks for editing/formatting some of the code bits I missed! The azure key vault code is a bit of a rabbit hole - but essentially it does an https oauth call to authenticate (and I manually set all keys via Env vars, so that's fine). I suspect maybe VSTS is blocking https calls. In any case - as you should know, we unfortunately live in a world of leaky abstractions - not all problems can be tested by pure unit tests. For now I've used the xUnit Trait attribute to bypass some of my integration tests — Adriaan de Beer, Jan 04 '18 at 20:36

score 3 · Answer 1 · answered Nov 09 '18 at 03:18

Just an update

I've recently set up a new xUnit .NET Core 2.0 project. For this project I had no issues passing environment variables to Azure DevOps. Basically, all you do is:

Create a Build pipeline variable - e.g. ASPNETCORE_ENVIRONMENT
Create Visual Studio Test Task
In your xUnit project, simply use System.Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT")

Given that the above works, .AddEnvironmentVariables() configuration method should just work fine now aswell. This mechanism can be used to load a different config files per test environments too.

For me, this is all I needed as I have a nice easy way to pass environment-specific options to integration tests

score 0 · Answer 2 · answered Jan 05 '18 at 05:42

0

The format of overriding test run parameters is like AppURL=$(DeployURL);Port=8080 instead of -name value

Supplying Run Time Parameters to Tests

answered Jan 05 '18 at 05:42

starian chen-MSFT

33,174
2
29
53

That's one of the many pages I looked at. Didn't work for me and seems outdated. The -key value was suggested by vsts vstest tooltip, as oice mentioned above. I suspect maybe this isn't implemented by .net core 2 test adapters yet... – Adriaan de Beer Jan 05 '18 at 10:12
I assume you are using Visual studio test task 2.* and the override parameter format is -key value, with this task, it will update the runsettings file before running the test, can you get the original value in runsettings file if you do test in local? – starian chen-MSFT Jan 08 '18 at 02:33
With Visual studio test task 1.*, it works fine with the parameter format like key=value. (Core 2.0 Unit test project) – starian chen-MSFT Jan 08 '18 at 02:34
Thanks @starianchen-MSFT Unfortunately the test task 2.* approach didn't work for me - the test keeps on using the original values (which is strange since you say it should replace the values in the source settings file before running the test. If I run 'dotnet test /?' locally it certainly seems to indicate that i should add ' -- webAppUserName=foo' as command line parm - but again, it doesn't work! – Adriaan de Beer Jan 11 '18 at 05:08
What's the result with test task 1.*? (format key=value) – starian chen-MSFT Jan 11 '18 at 05:36
1

Unfortunately, this issue is yet to be resolved in .net Core - there are several issues open about this in github [345](https://github.com/Microsoft/testfx/issues/345), [315](https://github.com/Microsoft/testfx/issues/315), [24](https://github.com/Microsoft/testfx/issues/24) For now, best advice I can give anyone requiring test parameters is to use a .net full framework test - not .net core – Adriaan de Beer Jun 28 '18 at 00:07

Passing KeyVault secrets to .net core 2 xUnit/MsTest in VSTS

2 Answers2