17

In JUnit you can use @Ignore before methods to tell the test runner to automatically skip those tests. From what I can gather, this is really only a convenient way to document/mark off incomplete/no longer functional tests that you want to get back to later.

Am I correct in saying then, that at runtime there is no difference between an @Ignore test, a method with no annotation, and a commented out method? (Assuming these tests are all self contained.) Is there some way to obtain a list of the ignored test cases in JUnit on Netbeans? If not, how much usefulness does the @Ignore tag really have, since it might be more useful to fail the test so it's not overlooked?

kaya3
  • 47,440
  • 4
  • 68
  • 97
donnyton
  • 5,874
  • 9
  • 42
  • 60

6 Answers6

24

Am I correct in saying then, that at runtime there is no difference between an @Ignore test, a method with no annotation, and a commented out method?

An @Ignored method can be found via reflection. A method with no annotation can't (or, to be precise, it can't be identified with certainty as an ignored test method), and a commented out method does not even get into the bytecode.

Albeit I don't think there would be much practical value in finding @Ignored methods runtime, it may be useful to generate statistics / reports.

how much usefulness does the @Ignore tag really have

One thing I can think of is searchability. You can easily identify all @Ignore annotations in the source code, while unannotated or commented out tests are not so simple to find.

it might be more useful to fail the test so it's not overlooked?

If you want (and can) fix it right away, it is fine to have it fail. There are cases when you can't, but you still want to method to be around, precisely so that it does not get forgotten. Then @Ignore makes sense.

Péter Török
  • 114,404
  • 31
  • 268
  • 329
9

  • A method with no annotation most likely is a utility method, likely used by other methods in the test.

  • A commented-out method is one that is completely hidden.

  • But you might use @Ignore to indicate that you are temporarily disabling the test. If you have a large and complicated piece of code that is in transition, you might want to temporarily disable some tests while that code is failing, especially if the mechanism running that code is run as part of some larger regression suite. The @Ignore attribute would be a reminder to you (or your QA team) that there is still more to do.

    Remember that @Ignore'd tests are still live code, and can be seen and even executed via reflection, which commented-out code cannot be. And it also means when you refactor other code, they are also updated, and syntax breakage in an API will show there. So even tests with @Ignore set can actively contribute to quality of code.

  • JUnit 4 TestRunners will show @Ignore'd tests as "skipped" so you can see in the end how many tests passed/failed/skipped.

Although your copy of Netbeans may not take full advantage of @Ignore, I can assure you that it is not the only consumer of JUnit tests.

lavinio
  • 23,931
  • 5
  • 55
  • 71
  • true, those are the implications of the different ways to disable a test. but at runtime they all become equivalent, right? (assuming it is not a utility method) – donnyton Jul 15 '11 at 13:00
  • 4
    Except for two things: **1.** `@Ignore`'d tests are visible via reflection, so there is some additional information conveyed there. **2.** If you comment out a test, you'll never know that you broke your API causing a potential syntax error. Keeping the code there and `@Ignore`'d means at least the syntax is still correct during code changes. – lavinio Jul 15 '11 at 13:06
  • 1
    @Ignore tests are also (or should be) reported by JUnit4 TestRunners as "skipped" tests - which is still useful information in reports. – Nate Jul 15 '11 at 15:26
  • @lavinio : you may want to integrate your 'except for two things' comment into the text of your answer... it would improve the answer. – vkraemer Jul 15 '11 at 17:29
6

@Ignored tests are more maintainable (and maintained) than commented-out ones. They are subject to compiler errors as things evolve, structured searching via IDE's, and IDE-assisted refactoring.

Ed Staub
  • 15,480
  • 3
  • 61
  • 91
3

FYI, TestNG supports this slightly differently:

  • You can include a description of why the test is being ignored:

    @Test(enabled = false, description = "Disabled until BUG-1234 gets fixed")
    • The Eclipse TestNG plug-in and the HTML reports will give you a list of all the tests that have been disabled, along with the reason why:

enter image description here

Cedric Beust
  • 15,480
  • 2
  • 55
  • 55
1

Using @Ignore instead of commenting out @Test has the advantage that Hudson/Jenkins recognizes this as an ignored test and displays it in the test results accordingly. So you get reminded about that ignored Test which is a big benefit!

Kai
  • 38,985
  • 14
  • 88
  • 103
  • On the flip side (and I suspect this may have prompted the post), someone using an IDE will immediately spot (by color) that a test is commented out, but may easily miss that it's being @Ignored. – Ed Staub Jul 15 '11 at 13:31
  • 1
    Thats true if you scroll through the class. But if you run the test class Eclipse will show you skipped tests by a crossed out icon in the results ;-) – Kai Jul 15 '11 at 13:35
0

IMHO, @Ignore's usefullness would be to know in the reports that there are x skipped tests, which you don't get if the test isn't annotated as a test or is commented. Failing the tests isn't a good option as we know that the tests we want to ignore aren't relevant at some point, but will have to be reenabled later. So we want to see if all the other tests will succeed/fail, knowing that we will skip the irrelevant ones (the fact that we've skipped them will be visible in reports).

Jihed Amine
  • 2,198
  • 19
  • 32
  • I've heard this, but where do I find the reports? Netbeans does not report it in the visual JUnit runner. – donnyton Jul 15 '11 at 13:05
  • I don't know about Netbeans, I use maven commands in a terminal (mvn test / integration-test). The reports are generated in target/failsafe-reports or surefire-reports by default but the destination folder can be customized. But in Netbeans, don't you get a summary with X Tests, X1 Failures, X2 Errors, X3 Skipped ? The number of skipped tests is inferred from the @Ignore annotations. – Jihed Amine Jul 15 '11 at 13:09