Robert Ramey wrote:
Quoting Beman:
Case in point, the doc build tool chain broke for this release, forcing a really messy workaround. Joel is trying to fix it now, but it is slow going since the test cases that fail for others works for him.
So, it appears like the problem is not that there's no *testing infrastructure* is missing, it's that for some behaviours, there are no tests or those tests are not sufficiently reliable.
Taking the case of boost book. As far as I know there is no regression testing to verify that it works when something it depends upon changes.
I don't know, either. But large part of the doc toolchain is outside of Boost -- e.g. Docbook and FOP. Those are huge, and it's not likely somebody will be able to create tests for those. So, manual inspection of results remain the only solution.
Therefore, it does not seem like generic advice of "we need tests" is going to help much -- there needs to be actual work on specific tests.
My suggestion is a lot more than "we need tests". My suggestion is that boost tools should be subjected to the same procedures that boost libaries are. Using boost book as an example, I would like to see boost/tools/boostbook/test/Jamfile ... and see that boost book shows up in the test matrix just like any library does.
I think that presence or lack of boost/tools/boostbook/test/Jamfile is an implementation detail.
I would like to see boost book "packaged" with documentation, examples and tests so that users would feel confident to use it for their own projects. If the "boost process" is good for libraries, I think it would be good for tools too. I would love to use boost book / quickbook for my own projects, but I don't feel its polished enough to depend upon.
I'm positively sure I saw documentation for boost.book. Quickbook is also documented. There's tools/quickbook/test/Jamfile.v2, and some test files there. For Boost.Build, there's both documentation and extensive test suite. So, it appears that your proposal boils down to: 1. Boost.Book should have tests 2. Test for tools should be included in the main test matrix Have I left out something?
The only argument I can see against this is that it seems to be alot of extra work. I'm sympathetic to this. I used to think that. But, the boost requirement for a separate test suite and the development of the serialization library made me realize I was wrong. I believe that extending the application of of the "boost process" to the tools would save effort in the long run.
And even if you add a new test for each issue encountered during release process, it does not automatically mean that the resulting release packages need not be examined. For example, here at work we do have very extensive automatic tests, but still, every user-visible package goes through manual QA -- and yes, it does checks that documentation exists.
No one has suggested that testing proves the absense of bugs.
I've understood your prior statement ("the issue raised above would not have occurred.") in exactly this way. I apologize if I've misunderstood what you was saying. - Volodya