Thu 23 Nov 2006
The Extreme Programming paradigm is a very interesting concept. I don’t agree with all of it and some of the things I do agree with are not always easy to achieve. One of the ideas I’ve absolutely fallen in love with, though, is the idea of integrated self tests, or “unit tests” as they’re sometimes called.
I have my own personal library of C++ code. It’s posted as free software under “starlib” on SourceForge, if you want to check it out… just don’t expect any real documentation. This software is an interesting study, I think, because it was originally written way back before the STL was available and the design has evolved as it has been used for different projects and operating systems. But I digress…
Until 2004, STAR had no built-in tests. When the idea was presented to me and I wanted to use that library in a new project, I decided to do yet another design upgrade and add support for self-tests. This was not easy! STAR is primarily an I/O abstraction library and testing I/O is difficult because at some point the operating system gets involved and then you simply can’t interfere. So I went to the base “tIoPath” class (a thin wrapper around open, close, read, write, etc.) and added hooks to all the functions there. Once you attach to the path, all reads and writes instead go to memory buffers from which the test case can eject and inject data. The test case can thus emulate what the operating system needs to do to exercise the code being tested. You can also set a callback function to intercept almost every I/O system call there is (including “read” and “write” if you’d rather interface directly instead of using intermediate memory buffers).
All this took a couple weeks for the initial version with numerous improvements/extensions over the next two years. It was a large piece of work. The benefits from it, though, are ten-fold! Now, any time I want to write a new I/O class, I can quickly and easily add test cases that will exercise all of code paths. I can inject bad data, cause data packets to get broken in odd places, and force write calls to only accept odd amounts of data at one time. These are all things that can happen when talking to the OS, but usually don’t, making it difficult to test any other way.
I say these test cases are easy, but that’s a relative term. They’re easy compared to what it would be like trying to do so without the underlying hooks in the I/O base class. They’re still work. However, all this needs to be tested anyway, so as a programmer I’d still have to find ways to create these situations by hand in order to verify the code I’ve written. That takes time, too. In the beginning, it would have been faster to hand-test things, but now the support is broad enough that I think writing test-case code is actually faster. But that is just the first time you test! The real benefit of built-in tests is that there is almost zero cost to performing it a second, third, fourth, or ninety-ninth time!
The key to self-testing is to make sure that all tests (or, at least, a significant subset) are run every time a build is done. If you change something, there is no need to go through the tedious hand testing again; the test cases written during the initial testing are still active and will validate your changes with no additional cost! That’s free work! You simply cannot afford not to have built-in self tests (appologies for the double-negative).
Some people argue that there should be a built-in test for every single method in a class and that all big methods should be broken down in to smaller, individually-testable methods. I don’t agree with that. I believe that a class should be tested completely at the API level, or “public interface”. If you’ve completly tested the public interface, then you’ve tested all the code of a class. If not, what is the purpose of that extra, untested code? Sure, you can test private parts if you wish; I just don’t think it’s critical. Plus, the API is less likely to change than the internal implementation so there are fewer changes required in the test code itself as maintenence is done.
If you’re still not sure exactly what I mean, try it. If you’re running a Unix-like system, try this:
cvs -d :pserver:starlib.cvs.sf.net:/cvsroot/starlib co star cd star ./configure make
At the end of the build, you’ll see it run it’s test suite. Better than that, it’s easy to incorporate this in any programs built using this library. You build your application and then just type ./myapp –test and !presto! you see all your tests run and have confidence that your app is ready to go. If you take the app to a new host, just run the test again to make sure that all required libraries are available; you don’t have to worry that it will fail at some unusual condition after running for an hour.
When it comes time to ship and you don’t want the bloat of the test cases, you can rebuild it all without the tests. It’s that easy!
I’m not saying all this to advocate my library. Far from it. It’s just an example of how useful this type of system can be. Once you’ve tried it, you’ll never want to go back to hand-testing again.
To summarize… Built-in self tests are a great way to validate your code as it’s written. The real benefit of it, though, comes when you’re doing maintenence! Whether it be bug-fixes or improvements, the built-in tests will ensure that your code is always working as it’s supposed to, and it will do so for free.