On Jun 2, 2010, at 12:36 PM, Beman Dawes wrote:

On Wed, Jun 2, 2010 at 1:41 PM, Belcourt, Kenneth <kbelco@sandia.gov> wrote:

...

Hi,

Could I suggest that when the trunk tests are completely broken, as they are now, and the report generation is cycling normally (every 1 to 2 hours) as it is now, that the likely source of breakage is a recent commit to the trunk repository. To reproduce the trunk failure, simply checkout out a current version of the trunk, cd to the status directory, and run bjam. If you do this right now, you'll find this error message.

error: Unable to find file or target named error: 'test_diamond_complex.cpp' error: referred from project at error: '../libs/serialization/test'

This is what's currently killing trunk testing.

bjam shouldn't stop entirely because a file isn't found. At worst, that library's tests should fail.

And all libraries that depend on the broken library will also likely fail.

Can Boost.Build or bjam be modified to be more tolerant of errors during regression testing?

Fair question. I suspect so but I've never looked into what it would take. -- Noel

Hi Rene, On Friday, 4. June 2010 06:05:12 you wrote:

It can be made to fail test if they are missing sources.. Since we already do that for dependent libs.

Could you please do it ? The tests are broken again right now due to two missing files from Boost.Config test. Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! Ingenieurgesellschaft für * voice: ++49 511 262926 57 ! Verkehrs- und Eisenbahnwesen mbH * fax : ++49 511 262926 99 ! Lister Straße 15 * juergen.hunold@ivembh.de ! www.ivembh.de * * Geschäftsführer: ! Sitz des Unternehmens: Hannover * Prof. Dr.-Ing. Thomas Siefer ! Amtsgericht Hannover, HRB 56965 * PD Dr.-Ing. Alfons Radtke !

Hi Rene, On Friday, 4. June 2010 16:53:02 you wrote:

On 6/4/2010 9:34 AM, Jürgen Hunold wrote:

...

Could you please do it ?

I can try :-)

That would be great.

...

The tests are broken again right now due to two missing files from Boost.Config test.

Ah, that seems to be working fine for me. What problem are you having?

Author: johnmaddock Date: 2010-06-04 08:37:44 EDT (Fri, 04 Jun 2010) New Revision: 62425 URL: http://svn.boost.org/trac/boost/changeset/62425 it is in libs/config/test/all/ bjam -j4 error: Unable to find file or target named error: '../no_0x_hdr_typeindex_pass.cpp' error: referred from project at error: '.' I simply touched them, so ? no_0x_hdr_typeindex_fail.cpp ? no_0x_hdr_typeindex_pass.cpp are missing. Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! Ingenieurgesellschaft für * voice: ++49 511 262926 57 ! Verkehrs- und Eisenbahnwesen mbH * fax : ++49 511 262926 99 ! Lister Straße 15 * juergen.hunold@ivembh.de ! www.ivembh.de * * Geschäftsführer: ! Sitz des Unternehmens: Hannover * Prof. Dr.-Ing. Thomas Siefer ! Amtsgericht Hannover, HRB 56965 * PD Dr.-Ing. Alfons Radtke !

...

It can be made to fail test if they are missing sources.. Since we already do that for dependent libs.

Could you please do it ? The tests are broken again right now due to two missing files from Boost.Config test.

Oh shucks, that would be me, fixed now, but it's all too easy to fall into this particular trap... John/

...

Could you please do it ? The tests are broken again right now due to two missing files from Boost.Config test.

Oh shucks, that would be me, fixed now,

John, I recently merged config trunk into release, so could you please do that for these fixes too as soon as your are sure they are stable? Thanks!

Nod. Will do, John.

Beman Dawes wrote:

On Wed, Jun 2, 2010 at 1:41 PM, Belcourt, Kenneth <kbelco@sandia.gov>

First of all, let me apologize for any inconvenience this may have caused.

bjam shouldn't stop entirely because a file isn't found. At worst, that library's tests should fail.

Can Boost.Build or bjam be modified to be more tolerant of errors during regression testing?

This is the perfect example of why I think trunk testing should be changed. Here is what I propose. For each library: - e.g. serialization a) Start with a machine which two separate release and trunk directory trees. Assume that these are named /boost_release and boost_trunk respectively. b) invoke the following script: mv boost_release/serialization boost_x/serialization mv boost_trunk/serialization boost_release/serialization mv boost_release/archive boost_x/archive mv boost_trunk/archive boost_release/archive mv boost_release/libs/serialization boost_x/libs/serialization mv boost_trunk//libs/serialization boost_release/libs/serialization We now have a boost_root tree which is the latest release but has the trunk for the one library being tested. c) run the test cd boost_release/libs/serialization/test bjam .... # I run ../../../tools/regression/src/library_test.sh ...which is equivalate to the above # but creates a report in the test directory c) run a script which reversed the changes made in a) above. The testing of all of boost would be defined as nothing more than the same procedure applied to each library. What would this give is? a) Problems such as the current one would be be limited only to the offending library. b) bugs in one library wouldn't show up as test failures in another library. c) Testing would be against the "next release". This would dimish surprises that might occur when libraries are merged into the release branch. d) Testing would be MUCH faster. Currently, if a header in library A changes in the trunk - then ALL the libraries which depend on A are rebuilt and tested. This costs a huge amount of time for almost no benefit. Under the new system, only the tests for library A would be re-run if library A changes. Tests for all other libraries which depend upon A would occur when library A is merged into the release branch. e) It takes a baby step in the direction of making boost libraries independently deployable. f) Certain libraries and tools such as bjam and boost.test are used in the testing process. When the developer of one of these tools checks in something that turns out to be an error - then it creates bogus test results when can hang around for weeks until they are cleared up. How can these developers work in such an environment? Finally, what would it take to implement this? Only a change in the test script. All current boost tools - bjam, boost test, etc would remain identical. I don't know how the test matrix generation works - but I suspect that it would change little if at all (I'm assuming it reviews the bin.v2 directory which would be the same under this proposal). Who sees anything wrong with this? Better yet, who can see any advantage in the current setup? Robert Ramey

Not me; everything under ryppl will (essentially) work that way. I hope we'll do that.

Speaking of ryppl, how ready is it, in your opinion, for prime time? I think I've had plenty of feedback on the proposed endian facility and would not mind being a guinea pig for developing a library from scratch under ryppl. Do you think I'm going to run into trouble getting things integrated with the boost SVN (well, that of course is making a huge assumption that the library will get accepted, but that's besides the point). Thanks.