When Is It OK for a Unit Test to be More Comprehensive?
I have a habit in our unit tests which is to write tests which are less granular than one might expect of a typical unit test.
My problem with granular tests is that I want the tests to be as useful as possible and I hate to repeat myself. So I tend to write systemic unit tests.
For example, we have nearly 150 classes which do image processing. Most of them are serializable. Rather than test them all individually, I wrote a test that reflects on our assembly and makes sure that any concrete class that inherits from ImageCommand and implements ISerializable will serialize and deserialize and produce an object which is identical (from a public interface point of view). Not only does this test hit everything that’s there, it will hit any future ImageCommands as well.
Since I’ve written several tests of this category, here is some advice to make your job easier when you write unit tests that are more system-wide:
- Use tests to generate expected data if it doesn’t already exist the first time and compare against it every time after. I do this for what I call stability tests: do my classes do the same thing always? If a class isn’t consistent, there’s a bug, and now I have a test.
- Accumulate errors, if possible, instead of failing immediately. This lets you get more per run of unit test.
- If you can, in your test output, generate the code for the individual test that’s failing. This is usually not much more work and will make it easier to skip the good results when you’re bug fixing.
- Don’t use random data; favor exhaustive data if you can. If you can’t be exhaustive, choose meaningful data.
- If you’re working with images or other things with a clear visual representation, on error write the results or the delta from correct into an output file and report the location of this file.
- NUnit allows you to put categories on tests. We use one called “Nightly” which violates the spirit of continuous integration of running all the tests on every build, but it allows continuous builds to finish in 20 minutes instead of hours (a trio of tests I just wrote take 4 minutes each).
Each of these is designed to reduce the cost of a failure and invite you to fix it instead of inducing you to ignore it.
Strongly recommended reading: The Way of Testivus.