1

I am trying to set up an automated tests using PHPUnit and Selenium with headless firefox. When Travis CI tries to run my tests, Selenium server fails to start, but my test is considered OK, because PHPUnit marks it as skipped:

The Selenium Server is not active on host localhost at port 4444.
OK, but incomplete, skipped, or risky tests!
Tests: 1, Assertions: 0, Skipped: 1.
The command "make test" exited with 0.

In my opinion, test should be considered as failed when it couldn't be even started for internal error. This is really stupid as my tests could fail this way and if didn't read the full report, I could believe everything is in a fact running okay, because Travis CI considers return value 0 to be successful test.

Is there an option to make PHPUnit return non-zero result when there are skipped tests? Or even make it directly report test as failed on Selenium error?

lot
  • 1,434
  • 18
  • 23

2 Answers2

3

Update:

See the accepted answer ( --fail-on-skipped ), added in version 9.4.3 ( about two years after the question was open )

Answer for previous versions:

Consider configuring the following parameters in your phpunit.xml file.

     stopOnError="true"
     stopOnFailure="true"
     stopOnIncomplete="true"
     stopOnSkipped="true"
     stopOnRisky="true"

Reference

If you want to use the commandline args equivalents are:

  --stop-on-error             Stop execution upon first error.
  --stop-on-failure           Stop execution upon first error or failure.
  --stop-on-warning           Stop execution upon first warning.
  --stop-on-risky             Stop execution upon first risky test.
  --stop-on-skipped           Stop execution upon first skipped test.
  --stop-on-incomplete        Stop execution upon first incomplete test.

For your case, you want to stop on skipped.

SIDENOTE For a test to be considered FAILED there must exist a failing assertion. Since skipped tests are not actually executed, you cannot consider them as failed, you should rely on the execution summary and make sure that no risky or skipped tests took place.

If you want your risky and warned tests to be considered as a FAILING TEST, you may use the following args:

  --fail-on-warning           Treat tests with warnings as failures.
  --fail-on-risky             Treat risky tests as failures.

Reference

If for any reason you would like to make PHPUnit return an exit code other than 0 (success), consider taking a look to How to make PHPunit return nonzero exit status on warnings

alariva
  • 2,051
  • 1
  • 22
  • 37
  • thank you. Stopping is not exactly what I wanted (wanted to mark test as failed), but I can live with that. I wonder how many times code with errors made it to production just because this silly feature. – lot Aug 14 '18 at 17:04
  • You are welcome. Please note that for a test to be considered **FAILED** there must exist a failing **assertion**. Since skipped tests are not actually executed, you (phpunit) cannot consider them as failed, you should rely (read or programatically interpret) on the execution summary and make sure that no risky or skipped tests took place. – alariva Aug 14 '18 at 17:17
  • Strange. In my opinion this should be exactly the opposite - test should not be considered SUCCESS unless there is successful assertion. I see testing as a way to prevent wrong code from making it to production and how can you prevent that if you don't run your tests at all? I will keep that in mind when I design my own test framework ;) – lot Aug 14 '18 at 17:33
  • So maybe my answer was incomplete. I hope `--fail-on-risky` comes in handy. Also see https://stackoverflow.com/questions/38303547/how-to-make-phpunit-fail-on-risky-tests – alariva Aug 14 '18 at 17:41
  • Thank you very much for your help, but I tested and it doesn't work. I am greatly disappointed with PHPUnit and especially it's author - after reading discussion on GitHub, it turned out that creator of PHPUnit himself strongly advocated for this (idiotic) behavior. It seems like there is no way to make PHPUnit report test failures and thus nobody should use it in CI environment. – lot Aug 16 '18 at 14:35
  • Are you sure? Can you pastebin an example output of a CI run withe these params on? – alariva Aug 16 '18 at 15:53
  • Still the same as in my original question. I think it is because there is no --fail-on-skipped option which would be what I need. --stop-on-skipped will stop the test, but not report a failure. --fail-on-risky does nothing, because my test is marked as skipped, not as risky – lot Aug 16 '18 at 16:05
  • There were people asking for the same thing on PHPUnit GitHub page, all brought down by the maintainer telling them that this is what he thinks is correct. Maybe he doesn't do CI, I don't know. – lot Aug 16 '18 at 16:07
  • I honestly don't think this is a dead end problem. What are the actual tests you are skipping during CI? Just selenium? – alariva Aug 16 '18 at 16:43
  • at the moment it is only the selenium test, but problem is general and will always be there unless PHPUnit author changes his opinion – lot Aug 16 '18 at 16:56
  • I get your point. However, I believe PHPUnit works like this because *this* (and other similar problems) can be addressed with little effort while designing the test suites. Please see https://stackoverflow.com/questions/3671685/how-to-run-a-specific-phpunit-xml-testsuite Try to build 2 suites so you *only* run the desired tests during CI. Secondly, considering that PHPUnit is OpenSource, you may perfectly fork and add your custom usage feature at your very own will (and very own risk). One of the benefits of OpenSource is ease of auditing, when everybody is wrong but me, better think twice. – alariva Aug 16 '18 at 20:16
  • forking might be a good idea. I don't get the point about building 2 suites - I already have 2, one is unit tests and other is selenium tests, but I do want to run them all in CI – lot Aug 17 '18 at 13:08
  • *The Selenium Server is not active on host localhost at port 4444.* Your problem is none of above. You need to run Selenium on CI, bit you don't have installed. So your actual question is: how do I install and run Selenium tests during CI. Go ahead for your research. – alariva Aug 17 '18 at 14:48
  • Checkout https://testingbot.com/support/other/travis-ci https://docs.travis-ci.com/user/gui-and-headless-browsers/ – alariva Aug 17 '18 at 15:10
  • Thank you for all your help, but you are not getting my point. – lot Aug 17 '18 at 16:46
1

Since version 9.4.3, PHPUnit has a --fail-on-skipped option:

$ vendor/bin/phpunit --help
(...)
  --fail-on-incomplete        Treat incomplete tests as failures
  --fail-on-risky             Treat risky tests as failures
  --fail-on-skipped           Treat skipped tests as failures
  --fail-on-warning           Treat tests with warnings as failures
(...)
BenMorel
  • 34,448
  • 50
  • 182
  • 322