End Of The Green Bar Paradise? The Overestimation Of Unit Testing In Distributing Computing

Unit Testing is great, integration / functional testing even better. Unit Tests are aimed to verify an isolated chunk of functionality, integration tests same thing in more realistic environment. They are coarser in general. However all of these techniques test the application in sequential way. A test is executed after a test. This is really odd in distributed environment. However this approach gets more and more popular - mostly because of the "metrics driven development". Its getting even worse - what I observe is the misuse of mocking to build an own perfect world and just abstract the invonvenient "truth" (legacy servers, deployments etc).

The problem with the sequential testing of a distributed system is mostly the misinterpretation of the results. The "Green Bar" (in Eclipse), or "Green Text" (in Netbeans :-)), is just the indicator, that all you sequential tests are passed. This is not enough for the most applications in general (except single-threaded fat clients). "Green Bar, 90% test coverage" projects can really fail. Some samples:

In JMS messages are only ordered, in sequential case. Under load, they are mostly unordered
Optimistic Lock Exceptions mostly happen in concurrent environment (only then, different threads (users) will access the same entity)
Deadlock will never occur during unit testing
Inconsistencies (like direct access to the EntityManager from a webapp) are only visible under load
In Unit Testing there is no difference between the transactions, and "auto commit" mode.
Memory Leaks - only visible during long running tests
Resource allocation (data source, jms factories, JCA connectors, pooled components) works mostly during unit testing

However if your are using Netbeans, you are save -> there is no "Green Bar" :-). So how to start? Checkout http://jakarta.apache.org/jmeter/, and wait until Q & A prescribe some load testing metrics :-).

Comments:

Hi Adam,

thanks for these insights. The link http://jakarta.jmeter.org is broken - did you mean to point to the jMeter main site?

Regards

Malte

Posted by Malte Schnack on June 17, 2008 at 01:55 PM CEST #

Unit testing can be multithreaded, so do not tests multithreaded code with sequential tests, use nut crackers to crack nuts and "May The Force Be With You"!
it's all not about "metric driven development", but about using proper tools for the corresponding tasks... and common sense in the first place ;-)

Posted by abch on June 17, 2008 at 02:18 PM CEST #

I guess you misunderstand the aim of unit testing. All the issues you mention are of course not targeted by unit testing as unit testing aims to test very small units, mocking dependencies. This mostly serves the purpose to deliver more robust units into the stage of integration testing letting this stage concentrate exactly on the problems you described above.

It is much easier to test how a class reacts on a exception being thrown from one of its dependencies in a unit test, that constructing it in an integration test scenario.

Regards,
Ollie

Posted by Oliver Gierke on June 18, 2008 at 12:05 AM CEST #

Hi Oliver,

exactly - I only liked to mention the fact the a Unit-Test is fine, but in a distributed environment load and robustness tests are at least as important, as an isolated test of a pease of functionality. However they are often underestimated and actually not performed,

regards,

adam

Posted by Adam Bien on June 19, 2008 at 11:00 AM CEST #