Re: [boost] Understanding the test matrix

The release test matrix seems to be telling me that there are many failures on different platforms, and in fact that there is no platform with no failures on all libraries, or libraries with no failures on all platforms. We just live with that, is that right?
Do all the failures get fixed before an actual release is made, or noted in the release notes?
I think the original philosophy was that it should work on mainstream, modern compilers/platforms, but not necessarily on your old/outlier, and here's a matrix of what's been tested. I'm okay with that. Also, a single unexpected regression failure marks the whole library as fail/yellow on the main matrix. And something like serialization not linking cascades to several other libraries showing failed (e.g., date_time on mingw32). It should be that way, but it shakes the casual observer's confidence in all of boost. All this to say, maximizing the green on that regression matrix is worth the work, and worth holding up a release, IMHO. Also, many fixes are not as simple as just looking at the test results for a compiler/platform you have little experience with and can't test first-hand. (E.g., Ticket # 4736 -- it's clear the author was looking for the failure, but the only thing that made progress was my running it in gdb on that particular platform.)
participants (1)
-
Jim Bell