3

How can the Maven Failsafe plug-in be made to exit with a non-zero exit status if any test fails? By default it seems to exit with a zero exit status no matter if tests succeeded or failed.

This is a problem when running integration tests from an editor (like Emacs) that checks the exit status of the build command: If it sees a zero exit status, it assumes executing the tests succeeded. If it sees a non-zero exit status, it flags that visibly to the user. This cannot be used if Maven Failsafe always just exits with a zero exit status and the user is forced to review the test execution logs to discover if tests failed, which is a time-consuming task.

The command used is: mvn test-compile failsafe:integration-test -Dit.test=FooIT

Markus Miller
  • 3,695
  • 2
  • 29
  • 33
  • You should use the maven-failsafe-plugin:verify goal..which handles that... – khmarbaise Oct 11 '16 at 18:39
  • khmarbaise: Thanks, that seems to have been exactly what I was looking for (`mvn test-compile failsafe:integration-test failsafe:verify`). Feel free to post it as an answer. – Markus Miller Oct 11 '16 at 18:43
  • Are you really calling this on command line? What not binding it correctly to the life cycle and simply calling `mvn verfiy`? – khmarbaise Oct 11 '16 at 18:51
  • 1
    khmarbaise: The integration tests are bound to the `verify` lifecycle phase as recommended. However, for a faster development cycle when developing integration tests, I want to execute only the integration tests and none of the heavy lifecycle phases that precede it, so I invoke the Failsafe goals directly. – Markus Miller Oct 11 '16 at 18:53
  • Why not running the IT from your IDE? – khmarbaise Oct 11 '16 at 18:54
  • I'm using a development environment (Emacs) that interfaces with the tests using processes, so invoking the `mvn` command is the way to go here. – Markus Miller Oct 11 '16 at 18:56
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/125438/discussion-between-valko-sipuli-and-khmarbaise). – Markus Miller Oct 11 '16 at 18:56
  • @khmarbaise You might want to run a flaky integration test over and over again to see whether or not it fails and under which conditions. That's hard to do in an IDE without changing test code and whatnot, but easy in bash: `let i=0; let failure=0; while true ; do let i++; time mvn failsafe:integration-test failsafe:verify -Dit.test=ResponsibleAttenderExpirerITest -Dparallel=classesAndMethods || let failure++; printf "\nFailed %d of %d runs\n" $failure $i; done` – oligofren Sep 17 '21 at 11:51
  • Why do you: `That's hard to do in an IDE without changing test code ` ? Why? Can't you run your IT in your IDE? – khmarbaise Sep 17 '21 at 12:04
  • @khmarbaise maybe they are running the tests in a CI/CD pipeline after a deploy to see if the deploy was successful? – RDM Nov 19 '21 at 13:35

1 Answers1

1

Just posting the answer from the comments for posterity:

Add the failsafe:verify goal

Typically mvn failsafe:integration-test failsafe:verify -Dit.test=MyITest

Example script for stats on a test

Example of running a flaky integration test over and over again while skipping all the other lifecycle tasks (given that everything has compiled already):

let i=0; 
let failure=0; 
while true ; do 
  let i++; 
  time mvn --quiet failsafe:integration-test failsafe:verify \
     -Dit.test=ResponsibleAttenderExpirerITest \
     -Dparallel=classesAndMethods -Dfailsafe.printSummary=false \
     || let failure++;   

  printf "\nFailed %d of %d runs\n" $failure $i; 
done
oligofren
  • 20,744
  • 16
  • 93
  • 180