Deploying with team build and psexec -> integration tests error

Question

I have a team build (upgrade template, tfs2010, msbuild) compiling and testing a WCF service. We use psexec and the exec task to remote install the service (wix installer) on the web server, prior to running an integration test suite against it. However sometimes our nightly build fails with a compilation error - can only see the first 1024 bytes and most of it is css styles. I've tried to delay the tests with sleeps, thinking it might be due to long JIT. However all 600+ integration tests fails. In the build log it seems that the exec task with psexec executes synchronously, as expected, and returns an exit 0. Could anyone come up with a reason to why this occurs now and then?

Answer 1

This doesn't sound like it's anything specific to TFS, msbuild or psexec -- it sounds like there may be an interim installation, configuration, or coding problem with the service. The point of CI and integration tests is to get early feedback on your process, and apparently something is wrong. The trick is drilling down into the problem and ruling out where the issue(s) reside.

Psexec claims the WiX deployment went fine, but did it? Are all files present? Were previous versions of the installation properly removed, or did they get upgraded correctly?

All 600 tests fail, but the tests don't contain the proper stack trace - can you reproduce the problem outside of the tests? Eg, when the tests fail can you emulate a test manually or run one of the existing tests with a debugger attached to see the same stack trace? One strategy might be to identify one or two specific tests that accurately validate the deployment -- run just these tests after the deployment and if these tests fail you should abort the build and then leave the server in a failed state for deeper analysis. this may require customization the build template but it may be worth the effort.

Can you add logging to the WCF service? Better logging to the tests?

Lastly, CI as mentioned previously is about early feedback. The general rule is, "If something is painful then you should do it more often." Focus on pain points, isolate them and iteratively improve them. When the pain diminishes, focus on other pain points. In your case, consider running your "nightly" build in a rolling fashion - you'll find your intermittent problem after a few runs rather than weeks.