Vladimir Prus wrote:
Robert Ramey wrote:
Taking the case of boost book. As far as I know there is no regression testing to verify that it works when something it depends upon changes.
I don't know, either. But large part of the doc toolchain is outside of Boost -- e.g. Docbook and FOP. Those are huge, and it's not likely somebody will be able to create tests for those. So, manual inspection of results remain the only solution.
I'm not advocating that tests be create for those things outside of boost. I'm advocating tests for those things in boost which depend on things outside of boost as well as the thing itself. We do the same thing with our libraries. We write code which depends up STL. In the course of testing ,we discover that we either made a mistake in using STL, or that various verstions of STL have differing interpretations of ambiguous points in some standard or have bugs in thier own.
My suggestion is a lot more than "we need tests". My suggestion is that boost tools should be subjected to the same procedures that boost libaries are. Using boost book as an example, I would like to see boost/tools/boostbook/test/Jamfile ... and see that boost book shows up in the test matrix just like any library does.
I think that presence or lack of boost/tools/boostbook/test/Jamfile is an implementation detail.
That is the essential difference in out view point.
I would like to see boost book "packaged" with documentation, examples and tests so that users would feel confident to use it for their own projects. If the "boost process" is good for libraries, I think it would be good for tools too. I would love to use boost book / quickbook for my own projects, but I don't feel its polished enough to depend upon.
I'm positively sure I saw documentation for boost.book. Quickbook is also documented. There's tools/quickbook/test/Jamfile.v2, and some test files there. For Boost.Build, there's both documentation and extensive test suite.
So, it appears that your proposal boils down to: 1. Boost.Book should have tests 2. Test for tools should be included in the main test matrix
Have I left out something?
Not much. The only thing I might add is that I would like to see the directory structure code/tests/headers, etc reflect the boost pattern for libraries. This would make it easy to verify that everything exists.
No one has suggested that testing proves the absense of bugs.
I've understood your prior statement ("the issue raised above would not have occurred.") in exactly this way. I apologize if I've misunderstood what you was saying.
Well, maybe I didn't say it right. The reason I chose this particular instance as an example was that I believe that the application of the "boost process" would have made a difference in this particular example. I didn't mean that that this would solve ALL problems. I believe that it would result in a net improvement in quality and productivity. Robert Ramey