[test] Recent changes cause infinite recursion in unit test suite?
The recent changes to Boost.Test are causing math/common_factor_test to fail on all gcc and Unix-like platforms: the program goes into infinite recursion inside Boost.Test: #107563 0x004641cf in boost::unit_test::ut_detail::auto_tc_exp_fail<gcd_test_suite::gcd_unmarked_int_test_id>::auto_tc_exp_fail (this=0x4af2d0, v=0) at ../../../boost/test/unit_test_suite_impl.hpp:282 #107564 0x004641ab in boost::unit_test::ut_detail::auto_tc_exp_fail<gcd_test_suite::gcd_unmarked_int_test_id>::instance () at ../../../boost/test/unit_test_suite_impl.hpp:287 #107565 0x004641cf in boost::unit_test::ut_detail::auto_tc_exp_fail<gcd_test_suite::gcd_unmarked_int_test_id>::auto_tc_exp_fail (this=0x4af2d0, v=0) at ../../../boost/test/unit_test_suite_impl.hpp:282 etc etc. Any ideas or fixes? Many thanks, John Maddock.
John Maddock wrote:
The recent changes to Boost.Test are causing math/common_factor_test to fail on all gcc and Unix-like platforms: the program goes into infinite recursion inside Boost.Test:
FWIW, Tru64/CXX is also affected. And quite a few boost.test tests are failing because of this, too.
#107563 0x004641cf in boost::unit_test::ut_detail::auto_tc_exp_fail<gcd_test_suite::gcd_unmarked_int_test_id>::auto_tc_exp_fail (this=0x4af2d0, v=0) at ../../../boost/test/unit_test_suite_impl.hpp:282 #107564 0x004641ab in boost::unit_test::ut_detail::auto_tc_exp_fail<gcd_test_suite::gcd_unmarked_int_test_id>::instance () at ../../../boost/test/unit_test_suite_impl.hpp:287 #107565 0x004641cf in boost::unit_test::ut_detail::auto_tc_exp_fail<gcd_test_suite::gcd_unmarked_int_test_id>::auto_tc_exp_fail (this=0x4af2d0, v=0) at ../../../boost/test/unit_test_suite_impl.hpp:282
etc etc.
Any ideas or fixes?
Revert the change to boost.test?
Many thanks, John Maddock.
Markus
"John Maddock" <john@johnmaddock.co.uk> wrote in message news:018301c80fd4$fe4c3630$0f241452@fuji...
The recent changes to Boost.Test are causing math/common_factor_test to fail on all gcc and Unix-like platforms: the program goes into infinite recursion inside Boost.Test:
#107563 0x004641cf in boost::unit_test::ut_detail::auto_tc_exp_fail<gcd_test_suite::gcd_unmarked_int_test_id>::auto_tc_exp_fail Fixed in 40076.
Gennadiy
Gennadiy Rozental wrote:
"John Maddock" <john@johnmaddock.co.uk> wrote in message news:018301c80fd4$fe4c3630$0f241452@fuji...
The recent changes to Boost.Test are causing math/common_factor_test to fail on all gcc and Unix-like platforms: the program goes into infinite recursion inside Boost.Test:
#107563 0x004641cf in boost::unit_test::ut_detail::auto_tc_exp_fail<gcd_test_suite::gcd_unmarked_int_test_id>::auto_tc_exp_fail Fixed in 40076.
Nope, now I get: "Test setup error: test tree is empty" Followed by test failure, Still frustrated yours, John Maddock.
"John Maddock" <john@johnmaddock.co.uk> wrote in message news:033d01c8100d$970b7970$b9f01b52@fuji...
Gennadiy Rozental wrote:
"John Maddock" <john@johnmaddock.co.uk> wrote in message news:018301c80fd4$fe4c3630$0f241452@fuji...
The recent changes to Boost.Test are causing math/common_factor_test to fail on all gcc and Unix-like platforms: the program goes into infinite recursion inside Boost.Test:
#107563 0x004641cf in boost::unit_test::ut_detail::auto_tc_exp_fail<gcd_test_suite::gcd_unmarked_int_test_id>::auto_tc_exp_fail Fixed in 40076.
Nope, now I get:
"Test setup error: test tree is empty"
Followed by test failure,
Can you give me an example. It looks like unrelated issue. Gennadiy
Gennadiy Rozental wrote:
"John Maddock" <john@johnmaddock.co.uk> wrote in message news:033d01c8100d$970b7970$b9f01b52@fuji...
Gennadiy Rozental wrote:
"John Maddock" <john@johnmaddock.co.uk> wrote in message news:018301c80fd4$fe4c3630$0f241452@fuji...
The recent changes to Boost.Test are causing math/common_factor_test to fail on all gcc and Unix-like platforms: the program goes into infinite recursion inside Boost.Test:
#107563 0x004641cf in boost::unit_test::ut_detail::auto_tc_exp_fail<gcd_test_suite::gcd_unmarked_int_test_id>::auto_tc_exp_fail Fixed in 40076. Nope, now I get:
"Test setup error: test tree is empty"
Followed by test failure,
Can you give me an example. It looks like unrelated issue.
Gennadiy, I've asked Eric Niebler to revert your recent changes. We can't tolerate this sort of wholesale breakage of the trunk. Please don't make any more changes to the trunk until we figure out why your testing is not detecting problems before you commit. --Beman
"Beman Dawes" <bdawes@acm.org> wrote in message news:47150346.5090604@acm.org...
Gennadiy Rozental wrote:
"John Maddock" <john@johnmaddock.co.uk> wrote in message news:033d01c8100d$970b7970$b9f01b52@fuji...
Gennadiy Rozental wrote:
"John Maddock" <john@johnmaddock.co.uk> wrote in message news:018301c80fd4$fe4c3630$0f241452@fuji...
The recent changes to Boost.Test are causing math/common_factor_test to fail on all gcc and Unix-like platforms: the program goes into infinite recursion inside Boost.Test:
#107563 0x004641cf in boost::unit_test::ut_detail::auto_tc_exp_fail<gcd_test_suite::gcd_unmarked_int_test_id>::auto_tc_exp_fail Fixed in 40076. Nope, now I get:
"Test setup error: test tree is empty"
Followed by test failure,
Can you give me an example. It looks like unrelated issue.
Gennadiy, I've asked Eric Niebler to revert your recent changes. We can't tolerate this sort of wholesale breakage of the trunk.
It took me all but 5 min after report to fix it and commit. Latest vertion of the trunk should've work (everything by John issue, which I can't reproduce). How can you expect me to make any changes? I am trying to be as responsive as possible. This is at best discouraging. I might have couple days now and than I might be busy for another half a year. These changes seat on my hard drive for a year now - I did not have time to invest to be able to track all the issues. Gennadiy
Gennadiy Rozental wrote:
"Beman Dawes" <bdawes@acm.org> wrote in message
Can you give me an example. It looks like unrelated issue. Gennadiy, I've asked Eric Niebler to revert your recent changes. We can't tolerate this sort of wholesale breakage of the trunk.
It took me all but 5 min after report to fix it and commit. Latest vertion of the trunk should've work (everything by John issue, which I can't reproduce).
The smoke tests run at revision 40093 still were showing a lot of failures. The new tests are running now.
How can you expect me to make any changes? I am trying to be as responsive as possible.
The expectation is that changes to a foundation library like test that is part of our testing infrastructure will be very well tested before they are committed to the trunk. We need to do a postmortem to figure out why your testing didn't determine that there was a problem before you did a commit. What tests did you run? On what platform? Did you add a new test to detect whatever failed? What other steps can we take in the future to prevent this wholesale breakage from happening again?
This is at best discouraging. I might have couple days now and than I might be busy for another half a year. These changes seat on my hard drive for a year now - I did not have time to invest to be able to track all the issues.
Yes, I found it very discouraging too. --Beman
Beman Dawes wrote:
Gennadiy Rozental wrote:
"Beman Dawes" <bdawes@acm.org> wrote in message
Can you give me an example. It looks like unrelated issue. Gennadiy, I've asked Eric Niebler to revert your recent changes. We can't tolerate this sort of wholesale breakage of the trunk. It took me all but 5 min after report to fix it and commit. Latest vertion of the trunk should've work (everything by John issue, which I can't reproduce).
The smoke tests run at revision 40093 still were showing a lot of failures. The new tests are running now.
How can you expect me to make any changes? I am trying to be as responsive as possible.
The expectation is that changes to a foundation library like test that is part of our testing infrastructure will be very well tested before they are committed to the trunk.
We need to do a postmortem to figure out why your testing didn't determine that there was a problem before you did a commit.
What tests did you run?
On what platform?
Did you add a new test to detect whatever failed?
What other steps can we take in the future to prevent this wholesale breakage from happening again?
This is at best discouraging. I might have couple days now and than I might be busy for another half a year. These changes seat on my hard drive for a year now - I did not have time to invest to be able to track all the issues.
Yes, I found it very discouraging too.
The Win32 smoke test at 40095 is now available. See http://mysite.verizon.net/beman/win32-trunk-results.html And reverting the Boost.Test changes resulted in a dramatic improvement. Dozens of tests that started to fail last night are now working again. It looks to me like Gennadiy should run the full test suite locally before committing changes to trunk. That would have certainly detected last night's snafu before it happened. --Beman
"Beman Dawes" <bdawes@acm.org> wrote in message news:47151E0B.4090506@acm.org...
The Win32 smoke test at 40095 is now available. See http://mysite.verizon.net/beman/win32-trunk-results.html
These tells me nothing.
It looks to me like Gennadiy should run the full test suite locally before committing changes to trunk. That would have certainly detected last night's snafu before it happened.
I already spend hours on testing before I do any commit. Waiting days for the full test to complete will make it impossible for me to make any changes. And it still will leave a lot of room for possible portability failures, since I can't run on all the *unix platforms. Gennadiy
Gennadiy Rozental wrote:
"Beman Dawes" <bdawes@acm.org> wrote in message news:47151E0B.4090506@acm.org...
The Win32 smoke test at 40095 is now available. See http://mysite.verizon.net/beman/win32-trunk-results.html
These tells me nothing.
Here is how you use the smoke tests: * Before committing changes that might affect other libraries, look at the smoke tests to see how many failures there are. I've added a failure count as the last row, so you don't have to count by hand. * Commit your change. (Do a single commit; one of the advantages of SVN is that it does commits atomically - all or nothing.) * Wait for the next smoke test. If your change affects a lot of libraries, that could be an hour. But if it only affects a few, the wait will be less than 20 minutes. Say 10 on average. * Look at the failure count. If it jumps up, you know your commit causes the failures. (You can verify that by the revision numbers, or by doing an svn log.)
It looks to me like Gennadiy should run the full test suite locally before committing changes to trunk. That would have certainly detected last night's snafu before it happened.
I already spend hours on testing before I do any commit. Waiting days for the full test to complete will make it impossible for me to make any changes.
For one compiler, a full VC++ test takes about two hours on a modern desktop machine. But an incremental test takes less; as short as 5 minutes for small changes. The changes you made last night caused my machine to take about 50 minutes to cycle through the full set of tests.
Most developers don't have to go to that extreme, but Boost.Test is special because so many of our developers depend on it.
And it still will leave a lot of room for possible portability failures, since I can't run on all the *unix platforms.
We aren't expecting every platform to be tested. But there is no excuse for committing code that breaks a lot of libraries for your personal desktop platform. And then no excuse for breaking Win32/VC++ or Linux/GCC for more than an hour or so, since we test those for you with tests cycling as often as every 20 minutes. --Beman
"Beman Dawes" <bdawes@acm.org> wrote in message news:47153D34.8070105@acm.org...
Gennadiy Rozental wrote:
"Beman Dawes" <bdawes@acm.org> wrote in message news:47151E0B.4090506@acm.org...
The Win32 smoke test at 40095 is now available. See http://mysite.verizon.net/beman/win32-trunk-results.html
These tells me nothing.
Here is how you use the smoke tests:
[...] When this procedure was esteblished? Is it described somewhere on site?
It looks to me like Gennadiy should run the full test suite locally before committing changes to trunk. That would have certainly detected last night's snafu before it happened.
I already spend hours on testing before I do any commit. Waiting days for the full test to complete will make it impossible for me to make any changes.
For one compiler, a full VC++ test takes about two hours on a modern desktop machine. But an incremental test takes less; as short as 5 minutes for small changes. The changes you made last night caused my machine to take about 50 minutes to cycle through the full set of tests.
My computer is not as powerfull. I would expect at least several hours if not longer for the full regressiontest run. And than multiply it by at least 6.
Most developers don't have to go to that extreme, but Boost.Test is special because so many of our developers depend on it.
And it still will leave a lot of room for possible portability failures, since I can't run on all the *unix platforms.
We aren't expecting every platform to be tested. But there is no excuse for committing code that breaks a lot of libraries for your personal desktop platform. And then no excuse for breaking Win32/VC++ or Linux/GCC for more than an hour or so, since we test those for you with tests cycling as often as every 20 minutes.
Well. I retested everything again. Gcc with mignw and cygwin, msvc 8.0 7.1, cw and intel. I'll give it another try and will follow procedure above. Gennadiy
on Tue Oct 16 2007, Beman Dawes <bdawes-AT-acm.org> wrote:
It looks to me like Gennadiy should run the full test suite locally before committing changes to trunk. That would have certainly detected last night's snafu before it happened.
If it's the case that Boost.Test changes routinely break tests for other libraries without breaking Boost.Test's own tests, it's a clear indication that the test suite for Boost.Test is incomplete. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
"Beman Dawes" <bdawes@acm.org> wrote in message news:4715181B.1000203@acm.org...
Gennadiy Rozental wrote:
"Beman Dawes" <bdawes@acm.org> wrote in message
Can you give me an example. It looks like unrelated issue. Gennadiy, I've asked Eric Niebler to revert your recent changes. We can't tolerate this sort of wholesale breakage of the trunk.
It took me all but 5 min after report to fix it and commit. Latest vertion of the trunk should've work (everything by John issue, which I can't reproduce).
The smoke tests run at revision 40093 still were showing a lot of failures. The new tests are running now.
Can't comment on these without seeing errors. I am running unit tests on NT with: msvc-8.0 msvc-7.1 cw intel-8.3 gcc-mingw All my tests passed before I commited last night.
How can you expect me to make any changes? I am trying to be as responsive as possible.
The expectation is that changes to a foundation library like test that is part of our testing infrastructure will be very well tested before they are committed to the trunk.
We need to do a postmortem to figure out why your testing didn't determine that there was a problem before you did a commit.
What tests did you run?
My unit tests
On what platform?
NT
Did you add a new test to detect whatever failed?
What failed? Issue with infinite recursion was bug in registration system. I did not notice it first bacause gcc fails silently on NT. But I did fixed last night not long after it was commited. :: before longjump is platform issue.
What other steps can we take in the future to prevent this wholesale breakage from happening again?
Independent library versioning would help. I can develop and see test results independently of anyone else. Gennadiy
[sorry if this posted more than once] On 16/10/2007, Gennadiy Rozental <rogeeff@gmail.com> wrote:
"John Maddock" <john@johnmaddock.co.uk> wrote in message > Nope, now I get:
"Test setup error: test tree is empty"
Found it. Fixed Locally.
FWIW, this error was the cause of most unexpected test failures that I was getting on MSVC9 (beta2). The tests that failed first time with this error worked second time around funnily enough, but that's irrelevant now. -- Darren
participants (6)
-
Beman Dawes -
Darren Garvey -
David Abrahams -
Gennadiy Rozental -
John Maddock -
Markus Schöpflin