On Wed, 24 Aug 2005 00:26:17 +0300, Peter Dimov wrote
Ben Artin wrote:
Can someone explain exactly why this wasn't caught in unit tests? I am not sure I understand why we missed this, but I am pretty sure it's important that we try to avoid this happening in the future.
The Dirxion tests:
http://engineering.meta-comm.com/boost-regression/CVS-HEAD/developer/smart_p...
are being run with inlining turned off.
The bug only manifests itself when inlining is on.
For what it's worth, there is a recognition of the issue of running regression tests under debug only (search the mail archive if you really want to read about it). And, in fact, we do now have some platforms running tests using release configurations where inlining should be enabled. I believe the 1.33 is the first Boost release to have achieved this level of testing -- so, in fact, I believe the level of Boost testing has never been higher.
Sure, bugs happen. My main point here is that we should make sure we learn something from them -- or, more importantly, that we should learn something from them *before* our users learn that Mac OS X releases of boost are not thoroughly tested.
What I'd learn is that Boost is getting ever bigger in size and more popular. So even if we are getting better at testing we might very well see 'more bugs' after a release because there's more code to have bugs and more people with more configurations to see them. This dynamic is likely to continue.
The lesson here is that we can't test everything and that we must make a beta or an RC available earlier. We've had similar configuration-specific bugs before, and they weren't caused by platform-specific implementations. Alpha testing simply can't obviate the need for beta tests.
I agree. Jeff