Adam Bien's Weblog
End Of The Green Bar Paradise? The Overestimation Of Unit Testing In Distributing Computing
Unit Testing is great, integration / functional testing even better. Unit Tests are aimed to verify an isolated chunk of functionality, integration tests same thing in more realistic environment. They are coarser in general. However all of these techniques test the application in sequential way. A test is executed after a test. This is really odd in distributed environment. However this approach gets more and more popular - mostly because of the "metrics driven development". Its getting even worse - what I observe is the misuse of mocking to build an own perfect world and just abstract the invonvenient "truth" (legacy servers, deployments etc).
The problem with the sequential testing of a distributed system is mostly the misinterpretation of the results. The "Green Bar" (in Eclipse), or "Green Text" (in Netbeans :-)), is just the indicator, that all you sequential tests are passed. This is not enough for the most applications in general (except single-threaded fat clients). "Green Bar, 90% test coverage" projects can really fail. Some samples:
- In JMS messages are only ordered, in sequential case. Under load, they are mostly unordered
- Optimistic Lock Exceptions mostly happen in concurrent environment (only then, different threads (users) will access the same entity)
- Deadlock will never occur during unit testing
- Inconsistencies (like direct access to the EntityManager from a webapp) are only visible under load
- In Unit Testing there is no difference between the transactions, and "auto commit" mode.
- Memory Leaks - only visible during long running tests
- Resource allocation (data source, jms factories, JCA connectors, pooled components) works mostly during unit testing
Posted at 09:50AM Jun 17, 2008 by Adam Bien in Java EE 5 Architectures And Idioms | Kommentare[4]
[This entry is based on / extends my books: Enterprise Architekturen, Leitfaden fuer effiziente Software-Entwicklung and: Java EE 5 Architekturen, Patterns und Idiome]
Hi Adam,
thanks for these insights. The link http://jakarta.jmeter.org is broken - did you mean to point to the jMeter main site?
Regards
Malte
Gesendet von Malte Schnack am June 17, 2008 at 11:55 AM CEST #
Unit testing can be multithreaded, so do not tests multithreaded code with sequential tests, use nut crackers to crack nuts and "May The Force Be With You"!
it's all not about "metric driven development", but about using proper tools for the corresponding tasks... and common sense in the first place ;-)
Gesendet von abch am June 17, 2008 at 12:18 PM CEST #
Hi Adam,
I guess you misunderstand the aim of unit testing. All the issues you mention are of course not targeted by unit testing as unit testing aims to test very small units, mocking dependencies. This mostly serves the purpose to deliver more robust units into the stage of integration testing letting this stage concentrate exactly on the problems you described above.
It is much easier to test how a class reacts on a exception being thrown from one of its dependencies in a unit test, that constructing it in an integration test scenario.
Regards,
Ollie
Gesendet von Oliver Gierke am June 17, 2008 at 10:05 PM CEST #
Hi Oliver,
exactly - I only liked to mention the fact the a Unit-Test is fine, but in a distributed environment load and robustness tests are at least as important, as an isolated test of a pease of functionality. However they are often underestimated and actually not performed,
regards,
adam
Gesendet von Adam Bien am June 19, 2008 at 09:00 AM CEST #