2

Running SpecRun from command line as part of a Continuous Integration setup, and recently an ignored (@ignore) test generated an exit code of 120 when SpecRun completed.

Currently, we break the build on any exit code not equal to 0 (universal success indicator!).

What does exit code 120 mean exactly? Is it simply "A test was ignored"? or does it imply more?

What other values are returned?

Sample build output below. We collect all exit codes (currently 1 SpecRun task, so only 1 exit code is collected), and print out to "exit codes"

58>  Done.
58>  Result: all tests passed (5 ignored)
58>    Total: 478
58>    Succeeded: 473
58>    Ignored: 5
58>    Pending: 0
58>    Skipped: 0
58>    Failed: 0
58>  
58>    Execution Time: 00:00:42.4550000
58>
58>  exit codes 120
58>
58>Build FAILED.
58>
58>Time Elapsed 00:00:44.21
========== Rebuild All: 57 succeeded, 1 failed, 0 skipped ==========
johnny g
  • 3,533
  • 1
  • 25
  • 40

1 Answers1

2

I found the list of error codes in the forum: https://groups.google.com/forum/?fromgroups=#!topic/specrun/vPQ7z2kpkbs

After ordering the list by the error code you get:

Unknown = 0,
Succeeded = 110,
Ignored = 120,
Pending = 210 
NothingToRun = 310,
Skipped = 320,
Inconclusive = 410,
CleanupFailed = 420,
RandomlyFailed = 430,
Failed = 440,
InitializationFailed = 450,
FrameworkError = 510,
ConfigurationError = 520,

Based on this list we use the following command line script on the build server to execute the tests:

SpecRun.exe [...]

if errorlevel 200 exit /b %errorlevel%

exit /b 0

We use the exit command to return the error code. For the higher error level codes (>= 200) we return the error code to break the build. However, error codes below that limit are treated as success and we return 0.

Using this limit (200) the Succeeded and Ignored cases are treated as successful and everything else breaks the build. If you also want to allow Pending tests (210) you can simply check for higher error code in the IF.

Tz_
  • 2,949
  • 18
  • 13
  • Thanks for the information. TechTalk shouldn't do this though since exit code 1 is pretty standard for some type of error in other applications. Which means if you write a script that is abstract and checks results of various exe's, then the special situation for specrun has to be dealt with. Yuck. – Subjective Reality Jun 20 '18 at 00:22
  • With SpecFlow+Runner 3.0 we changed the exit codes of the specrun.exe. You can find the actual ones here: https://specflow.org/plus/documentation/Test-Execution-Result-Codes/ – Andreas Willich Mar 23 '20 at 16:35