1

At my office we have a dispute regarding the necessity of unit tests in addition to integration tests for the classes that have the main responsibility of interacting with a filesystem (DB, etc).

The integration tests we have, are almost unit tests, as the tested object doesn't interact with other objects at all. The only reason, why we call the tests integration, is that the real filesystem is used in tests. And it is proposed to make the tested class use filesystem layer component, then mock it in tests (so we will call them unit tests), and check interaction with this component, rather than real filesystem results. Necessity of this change is what we discuss.


One point of view we have, is that unit tests are always required, because:

  1. Writing unit tests makes code much better
  2. Having unit tests, you don't need to care about real filesystem and side-effects of files, appearing at wrong locations
  3. A developer can fully test results by making the tested class use filesystem mock and setting proper expectations for that mock
  4. It is ok to tie the mock expectations to the specific internal algorithm of the tested class, because we do white-box testing with unit tests

Thus, the unit tests must always be written for such a tested class. And a filesystem layer component must always be used by that class for the purpose of testing.


Another point of view, is that unit tests are not needed for specific edge cases of classes that are devoted to filesystem interactions, because:

  1. It is not possible to properly verify, that a tested class works, just by having simple mock instead of real filesystem (or its full emulation). Filesystem is such a complex component, that:
    • A tested class can work in many different ways in order to achieve successful result. The mock expectations cover just one-two possible scenarios, so a unit test erroneously shows failures for a class that properly implements good algorithm, which is different from the one expected.
    • A tested class can work in a way, detected by mock as a successful scenario, while the class still does not produce right result. This can be because of quite complex reasons in real filesystem. All these reasons are impossible to be covered just by a mock.
  2. A unit test with mock and expectations is very fragile, because it is very tied to the tested class's internal algorithm. And the test erroneously fails upon even right changes of the algorithm.
  3. The integration testing is a proper and full replacement for unit testing for a case, when the class has just 1-2 public methods, and only dependency is a filesystem. Integration testing gives same benefits as unit testing for this case - clear dependencies, more readable code, etc.

Thus the unit testing with filesystem mocking is not needed in our case. It is fragile and is not accurate for this particular case of classes.


So, to sum it up, the question is:
Is integration testing fully enough for an edge case of having not a complex class, which has main responsibility to work with a filesystem (DB, etc.)?

The only difference between integration and unit tests for this class, is that with unit tests the filesystem mock would be used (class would be fully isolated), while with integration tests the real filesystem is used.

I would appreciate, if you can add the references to classic books, or maybe articles / presentations of well-known industry people, so we can have a really strong ground to support the resulting conclusion.

Andrey Tserkus
  • 3,644
  • 17
  • 24
  • http://www.growing-object-oriented-software.com/ this is one of the books that I refer for TDD, maybe this would be helpful. And by the way unit tests do not need to test each and every branch but rather focus on testing the behavior. – Narendra Pathai Mar 26 '13 at 05:42
  • http://stackoverflow.com/questions/129036/unit-testing-code-with-a-file-system-dependency this is one of the questions on SO that is similar to this. – Narendra Pathai Mar 26 '13 at 05:45
  • Thank you, @NarendraPathai. I've seen that ticket, however I'm also seeking for some references to generally adopted sources (books, articles, etc). – Andrey Tserkus Mar 26 '13 at 16:48

2 Answers2

1

The short answer here is yes, you could fully test a class with 'integration' tests. The better question, though, is should you do so?

I think you're getting too hung up on the difference in definitions between a 'unit test' (no outside dependencies) and an 'integration test' (has such dependencies). The goal with testing is to give you confidence that your code is working at all times, while keeping the associated costs of having that confidence down. So your question

Is integration testing fully enough for an edge case of having not a complex class, which has main responsibility to work with a filesystem (DB, etc.)?

is somewhat incomplete.

The most useful part of that distinction between 'unit' and 'integration' for our discussion is this: unit tests are easier and cheaper to write, maintain, and run.

To write a unit test, you just need to know the code. If a unit test fails, you know it's because of changes to the code. Writing an integration test requires setting up dependencies, eg creating files with specific contents, inserting rows in to a database, etc. If an integration test fails, it could be your code, or it could be your dependencies. For these reasons and others, integration tests are more complex, and therefore expensive, to create, maintain and run.

That increased expense should push the developer to separate classes encapsulating business logic from classes that handle interaction with outside systems, in an effort to minimize the number of integration tests required. The business logic can be tested with unit tests, which are cheaper.

Edit

It is possible that your class has complicated logic that itself is complicated because it has to handle complicated behavior in the underlying external dependency (ie, the file system in question). In that case, mocking the file system may be quite difficult in itself, and it may be cheaper/easier to just use a properly set up file system and write 'integration' tests.

The important point to keep in mind is what you're trying to achieve: confidence at a acceptable cost. If 'integration' tests are cheap enough, great. If you can get the same confidence cheaper using 'unit' tests, even better. The exact mix depends on the problem at hand.

sharakan
  • 6,821
  • 1
  • 34
  • 61
  • Thank you, @sharakan. I hear you, thus I added the clarification to the question, that the class has no external dependencies. The integration tests are called "integration" currently only because they use real filesystem. The difference from proposed unit tests, would be just to do everything through some filesystem object, which would be mocked during unit testing. – Andrey Tserkus Mar 26 '13 at 16:52
  • The class has no external dependencies, but the integration test uses a real filesystem? By most definitions, "a real filesystem" *is* an external dependency. – sharakan Mar 26 '13 at 17:49
  • From your edits, I think what you're saying is that the class under test is fairly complicated BECAUSE the behavior of the underlying file system is complicated, and has to be handled. That could change the cost/benefit analysis, I'll update my answer. – sharakan Mar 26 '13 at 17:53
0

It would be preferable to have a known state of the filesystem or DB for the tests. As an example, you do not want to have a test fail because it is trying to insert a record that already exists. This failure is not due to the code but a problem with the DB. Same thing can happen in the filesystem. However, you should write the best test that you are able to. If you can't easily mock the filesystem or whatever, then interact with it. Just realize that if the test fails it may not be a problem with the code.

An ugle test is better than no test. --The Way of Testivus http://www.artima.com/weblogs/viewpost.jsp?thread=203994

Now even if you do have tests with mocks, that does not mean that you should not have QA or some sort of integration test to make sure that everything connects correctly. I view that unit tests are for verifying that the internals of the code works correctly and integration tests tell me that all the pieces work together.

I don't know what language you are using but the documentation for PHPUnit gives some ideas about testing the DB and filesystem.

A unit test with mock and expectations is very fragile, because it is very tied to the tested class's internal algorithm. And the test erroneously fails upon even right changes of the algorithm.

For testing with mocks, you should not be tying to the algorithm. All that you are testing for is the expected behavior of the class. Not how it goes about doing it.

Schleis
  • 41,516
  • 7
  • 68
  • 87
  • Thanks, @Schleis. I hear you. Regarding not tying to the algorithm - what we experience is that it is impossible for filesystem mocks. The filesystem is a complex component, thus there is no way to properly put expectations for mocks without knowing algorithm in advance and expect something like "file at location /foo/bar is opened, some particular content is written to this file, file is closed, file is copied to /baz/bar, etc.". And such fragile and not full expectations are our main concern. – Andrey Tserkus Mar 26 '13 at 16:57