How to programmatically ignore/skip tests with ScalaTest?

Question

I am running some tests with ScalaTest which rely on connections to test servers to be present. I currently created my own Spec similar to this:

abstract class ServerDependingSpec extends FlatSpec with Matchers {

  def serverIsAvailable: Boolean = {
    // Check if the server is available
  }
}

Is it possible to ignore (but not fail) tests when this method returns false?

Currently I do it in a "hackish" way:

"Something" should "do something" in {
  if(serverIsAvailable) {
    // my test code
  }
}

but I want something like

whenServerAvailable "Something" should "do something" in { 
  // test code
}

or

"Something" should "do something" whenServerAvailable { 
  // test code
}

I think I should define my custom tag, but I can only refer to the source code of in or ignore and I do not understand how I should plug in my custom implementations.

How should I accomplish this?

score 38 · Answer 1 · edited Jul 29 '20 at 07:14

38

You can use assume to conditionally cancel a test:

"Something" should "do something" in {
  assume(serverIsAvailable)

  // my test code
}

or you can test for some condition yourself, and then use cancel:

"Something" should "do something" in {
  if(!serverIsAvailable) {
    cancel
  }

  // my test code
}

edited Jul 29 '20 at 07:14

conny

9,973
6
38
47

answered May 06 '16 at 07:08

Holger Brandl

10,634
3
64
63

2

Assume is provided for this as well: `assume(server.isAvailable)` – Eric Zoerner May 11 '16 at 21:14
Nice, I've adjusted the solution to show a full example. – Holger Brandl May 12 '16 at 06:58
should be without the !. I tried editing it but alas edits have to be at least 6 characters. – Eric Zoerner May 12 '16 at 21:17
Thanks for the hint. fixed. – Holger Brandl May 13 '16 at 11:34
The outcome of a failed `assume` is a `org.scalatest.exceptions.TestCanceledException` and the whole suite ending up being aborted, and in consequence the whole build fails. Not sure how is that different than `fail` (other than slightly different text output) – Grzegorz Oledzki Sep 22 '20 at 19:48
assume won't ignore/skip the tests, it will fail the whole unit test on CI/CD. – Maziyar Jan 08 '21 at 14:47
`cancel` too will terminate with an error message. `if(serverIsAvailable)` will skip the tests without error. Which is want as well. In my case I have a demo mode and when I compile for demo certain tests won't be needed. – Martin Sep 29 '21 at 16:45

Andreas Neumann · Accepted Answer · 2015-09-08T15:37:59.750

11

You can use Tags to achieve this:

Documentation on how to use Tags : http://www.scalatest.org/user_guide/tagging_your_tests

Adding and removing tagged test with command line parameters: http://www.scalatest.org/user_guide/using_the_runner#specifyingTagsToIncludeAndExclude

Example Code:

import org.scalatest.Tag

object ServerIsAvailable extends Tag("biz.neumann.ServerIsAvailable")

"Something" should "do something" taggedAs(ServerIsAvailable) in { 
  // your test here
}

Running the tests

Running the tests is a bitt tricky. It only works for testOnly and testQuick not test. In the example testOnly is short for testOnly *

 sbt "testOnly -- -l biz.neumann.ServerAvailable"

edited Sep 08 '15 at 15:37

answered Sep 08 '15 at 14:36

Andreas Neumann

10,734
1
32
52

I found this too, but the documentation only refers how to do it on the command line. However, I am running `sbt test` on Jenkins, and I want to programmatically skip the test when the server is not available. – rabejens Sep 08 '15 at 14:45
so should the program react when the server is available or do you want to configure Jenkins to infer if the server is available and then have jenkins adapt to run the tests accordingly ? – Andreas Neumann Sep 08 '15 at 15:05
Oh, now I understand. Currently I start and shutdown the test server programmatically (it is done with Docker on a test server by SSH'ing in there), but using Jenkins for this is the cleaner solution. – rabejens Sep 08 '15 at 15:14
1

Regarding the edit: I use `testQuick` anyway. Thanks! – rabejens Sep 08 '15 at 16:12

score 10 · Answer 3 · answered Aug 08 '16 at 16:59

10

Here is some trick to skip a test based on a condition:

object WhenServerAvailable extends Tag(if (serverIsAvailable) "" else classOf[Ignore].getName)

"Something" should "do something" taggedAs WhenServerAvailable in { ... }

answered Aug 08 '16 at 16:59

Dmitry

121
1
3

How to programmatically ignore/skip tests with ScalaTest?

3 Answers3

Linked