6

It's possible, with cabal, to set up a continuous build that records test successes/failures in a format many CI systems will accept with a command like:

cabal test '--test-option=--jxml=test-results/$test-suite.xml'

The important part here is that $test-suite is replaced with the name of the test, so that different tests put their results into different files.

When I use stack, all tests get literally the option --jxml=test-results/$test-suite.xml passed to them, so the end result is that the tests overwrite each others' results.

Is there any way to run all my tests with stack so that I can tell each test suite a different place to write their results?

I'd even accept some stack command that parsed the cabal file for me and told me what test suites there are, because then I could script a loop in bash calling each test one at a time.

Daniel Martin
  • 23,083
  • 6
  • 50
  • 70

2 Answers2

1

I'd even accept some stack command that parsed the cabal file for me and told me what test suites there are, because then I could script a loop in bash calling each test one at a time.

If you're willing to stoop to that, stack ide targets will return a list of targets, from which you can do some bashing to get the list of test suites. Something like this:

stack ide targets 2>&1 |
  while IFS=: read pkg type suite; do
    [[ $type = test ]] && stack test ":$suite" --test-arguments="--jxml=test-results/$suite.xml"
  done
user11228628
  • 1,526
  • 1
  • 6
  • 17
-1

stack test --test-arguments="--jxml=test-results/$test-suite.xml"

palik
  • 2,425
  • 23
  • 31
  • When I do that, the effect is that all my tests write to the same file, called `test-results/$test-suite.xml`. – Daniel Martin May 03 '19 at 14:12
  • Well, actually, using that syntax they all write to `test-results/-suite.xml`, because bash, but if I single-quote that bit you've got double-quoted I get all the tests writing to `test-results/$test-suite.xml`. Also, I mentioned this in my question. – Daniel Martin May 03 '19 at 14:16
  • something like that `XX="--jxml=test-results/$test-suite.xml" YY="stack test --test-arguments=$XX" $YY`. It would be wise to rename `test-suite`. [See](https://bash.cyberciti.biz/guide/Rules_for_Naming_variable_name) – palik May 03 '19 at 14:26
  • But the *ACTUAL POINT*, in any case is that unlike cabal (which will substitute `$test-suite` for the test name when it runs each test), stack will do no such substitution. So I still get all of the tests writing to the same file, and overwriting each other's results – Daniel Martin May 03 '19 at 14:36