2

Before you jump to an answer, let's define what I mean (note that you may have different definitions, and that's part of the problem, but this is what I'm using)

mock testing aka behavior-based testing--- tests the code does the right thing i.e., tests verify behavior. All collaborators are mocked.

unit tests --- low-level tests focusing on a small part of the system (like a class). When we use mock testing, collaborators are mocked.

integration tests --- test the interaction of two or more parts of the system (like two classes). The components under test are not mocked.

system tests --- test the system as a "black box" i.e., from the perspective of a user who does not have access to the internals of the system. Real components are used (database, http, etc)

What I'm slowly realizing is that when unit tests are done this way, you may not need integration tests.

  • The behavior-based unit tests should verify that components talk to each other correctly
  • system tests should catch bugs from using real components

Integration tests then become an optional troubleshooting tool when a system test fails (since they're more fine-grained). (However, you might argue that system tests with good logging are enough except for the occasional edge case.)

What am I missing?

Update: By "enough", I mean that these unit tests + system tests will catch all the bugs that unit + integration + system tests will find.

Update: By "enough", I mean are there bugs that unit + integration + system tests will find that unit + system tests won't find? What I'm really looking for is an example that shows integration tests are necessary.

Community
  • 1
  • 1
U Avalos
  • 6,538
  • 7
  • 48
  • 81
  • 3
    There is no such thing as "enough". You test as much as you can given the time allotted. Integration tests are often useful as a replacement for people pressing buttons. Or to hit places where implementation side effects or assumptions are likely to cause issues as code changes. –  Dec 31 '14 at 19:55
  • I'm surprised by the number of people blindly hitting upvote on the comment. "People pressing buttons" is what I call a system test but apparently no one bothered to read the question. Ditto "implementation side effects" will be caught by system tests (but integration tests will tell you *where* those are). – U Avalos Jan 01 '15 at 00:36
  • They're probably responding from the first comment. As for definitions, they vary, apparently. As for your expectation of what tests can accomplish. They aren't magic nets, hauling in every possible bug. I wish they were... –  Jan 03 '15 at 20:36

3 Answers3

0

What I'm slowly realizing is that when unit tests are done this way, you may not need integration tests.

Generally system tests (as you define them), when automated, will run much more slowly than integration tests. If you have automated integration tests that verify the same things as automated system tests, the integration tests should fail (or succeed) faster.

So it depends on your definition of "need". Having overlap between the different categories of tests can't hurt, and can provide value if they help you discover bugs faster. It depends on the magnitude of value I suppose (what is the ROI for having both).

danludwig
  • 46,965
  • 25
  • 159
  • 237
0

Here is a possible answer. (I honestly think that this is a complex question with a real answer of "it depends"). The answer according to this video is that integration tests aren't necessary (well, sort of...the title is misleading):

"J.B. Rainsberger - Integrated Tests Are A Scam"

http://vimeo.com/80533536

The reasoning behind this answer is that

  • isolated unit tests (unit tests where are collaborators are mocked) should really be collaboration/contract tests.
  • in the collaboration-test part, you 1) validate expectations, 2) validate that the component handles answers from its collaborators correctly.

Ex: the object under test is an address book that uses a special address-collection object to store a list of address. (OK, this is a really dumb example.) In 1), you need to verify that the address book calls the collection when its supposed to (the expectations). In 2), you need to verify the address book works under the different collection states---no addresses, 1 addresses, some addresses, many addresses, etc. (In practice, you would probably limit these to business use cases.) This is a state-based test usually using a stub for the collaborator.

  • in the contract-test part, you add tests for every collaboration test. The object under test is now the address-collection object. For 1), expectations, can it answer the questions? For 2), the answer-handling, can it answer correctly?

  • every object has collaboration tests for its collaborators. Every collaborator has matching contract tests but also its own set of collaboration tests for its collaborators. This forms a "ring model" of the architecture. The "outer layer" talks to external services (database, http, etc). You use integration tests for these.

So to answer my question, the short answer is "no". Integration tests are brittle and often complex. However, instead of eliminating them completely, a better strategy seems to be to minimize them.

Aside: this collaboration/contract method of testing may have its own set of issues. Matching tests are coupled together---a change in one requires a change in another. The use of a stub can lead to false positives. Intuitively, it seems that this type of testing has a non-trivial maintenance cost.

U Avalos
  • 6,538
  • 7
  • 48
  • 81
0

first of all, i don't see the difference between what you described as unit-tests and mock-tests.

as @danludwig suggested, it depends on your ROI. if you're writting short script for download all movies from youtube's playlist then you probably don't need any tests at all. when you do world wide shop for compulsive shopping of plane control system them what you mentioned is definitely not enough (degradation tests, failure tests and probably many many more).

from my experience in statistical web application, if you have a lot of unit-tests then you don't need many integration tests. sometimes you will foresee potential integration problems and you write required tests and sometimes your testers will find problems on UAT and then you will add required tests. but still number of integration tests will be far less than unit tests

regarding black box testing: if your application interacts with humans then you must have testers for manual tests (css, pdf rendering etc). they will also discover errors in communication between tested units (for new integration tests)

thing to remember:

  • important part of black box testing is also penetration testing
  • and remember about performance tests
piotrek
  • 13,982
  • 13
  • 79
  • 165