IIUC, we've already found some regressions with 1.31.0, having to do with the graph library and MSVC6. These regressions affect the Python lib also, and the problems show up in the regression test results. It's very disappointing to me that even though the Python library was being tested continuously, we let the release go out anyway. We ought to ask ourselves how this happened and see if there's something we can do to prevent it in the future. My guess is that inspecting regression results by hand isn't good enough, and we need an automated system like the BuildBot. We also ought to be thinking about releasing 1.31.1 ASAP. -- Dave Abrahams Boost Consulting www.boost-consulting.com
On Tue, 10 Feb 2004, David Abrahams wrote:
We also ought to be thinking about releasing 1.31.1 ASAP.
Then we need to pull together a (short!) list of the fixes we want in 1.31.1. Graph and Python fixes, as you've said. I wouldn't mind getting the /CLR workaround into Function. Doug
Douglas Paul Gregor <gregod@cs.rpi.edu> writes:
On Tue, 10 Feb 2004, David Abrahams wrote:
We also ought to be thinking about releasing 1.31.1 ASAP.
Then we need to pull together a (short!) list of the fixes we want in 1.31.1. Graph and Python fixes, as you've said. I wouldn't mind getting the /CLR workaround into Function.
I don't think there's a specific fix needed for Python, once the graph library is fixed. -- Dave Abrahams Boost Consulting www.boost-consulting.com
Hi Guys, On Feb 11, 2004, at 8:43 AM, David Abrahams wrote:
I don't think there's a specific fix needed for Python, once the graph library is fixed.
I fixed a VC6/python related bug a few days ago... are there any others? -Jeremy P.S. I fixed it in CVS head, that is. Should the fix be merged to RC_1_31_0? _______________________________________________ Jeremy Siek <jsiek@osl.iu.edu> http://www.osl.iu.edu.edu/~jsiek Ph.D. Student, Indiana University Bloomington C++ Booster (http://www.boost.org) Office phone: (812) 856-1820 _______________________________________________
Jeremy Siek <jsiek@osl.iu.edu> writes:
Hi Guys,
On Feb 11, 2004, at 8:43 AM, David Abrahams wrote:
I don't think there's a specific fix needed for Python, once the graph library is fixed.
I fixed a VC6/python related bug a few days ago... are there any others?
-Jeremy
P.S. I fixed it in CVS head, that is. Should the fix be merged to RC_1_31_0?
Absolutely, and ASAP, thanks! Can you say anything about the origin of the bug, like when was it introduced, and how come it didn't trigger any alarms in the regression testing? -- Dave Abrahams Boost Consulting www.boost-consulting.com
Hi Dave, On Feb 11, 2004, at 11:16 AM, David Abrahams wrote:
P.S. I fixed it in CVS head, that is. Should the fix be merged to RC_1_31_0?
Absolutely, and ASAP, thanks!
Ok.
Can you say anything about the origin of the bug, like when was it introduced, and how come it didn't trigger any alarms in the regression testing?
My best guess is that it was triggered by the fix of a performance bug in remove_edge. As for triggering alarms... do we have alarms that get triggered automatically? Or do you mean, why didn't someone notice that the regression for Boost.Python on VC6 started failing? Well, perhaps we should ask whose responsibility it is to watch the Boost.Python regression logs for VC6. Cheers, Jeremy _______________________________________________ Jeremy Siek <jsiek@osl.iu.edu> http://www.osl.iu.edu/~jsiek Ph.D. Student, Indiana University Bloomington C++ Booster (http://www.boost.org) Office phone: (812) 856-1820 _______________________________________________
Jeremy Siek <jsiek@osl.iu.edu> writes:
Hi Dave,
On Feb 11, 2004, at 11:16 AM, David Abrahams wrote:
P.S. I fixed it in CVS head, that is. Should the fix be merged to RC_1_31_0?
Absolutely, and ASAP, thanks!
Ok.
Can you say anything about the origin of the bug, like when was it introduced, and how come it didn't trigger any alarms in the regression testing?
My best guess is that it was triggered by the fix of a performance bug in remove_edge.
Ah, I remember those.
As for triggering alarms... do we have alarms that get triggered automatically? Or do you mean, why didn't someone notice that the regression for Boost.Python on VC6 started failing?
That's part 1. Part 2 is "why didn't a graph library test start failing as well?"
Well, perhaps we should ask whose responsibility it is to watch the Boost.Python regression logs for VC6.
I doubt there's any assigned responsibility. I was watching, but then I had to travel and lost connectivity. I did expect that Beman was going to look things over and make sure there were no new regressions before the release went out. I hypothesize that part of the problem is that he's not looking at the meta-comm tests, which include Boost.Python and Spirit and show regressions against the previous release, rather than just the last test run. I have been worried for some time that test effectiveness is diluted by having two reporting/display systems... did it bite us here? -- Dave Abrahams Boost Consulting www.boost-consulting.com
At 03:51 PM 2/11/2004, David Abrahams wrote:
Well, perhaps we should ask whose responsibility it is to watch the Boost.Python regression logs for VC6.
I doubt there's any assigned responsibility. I was watching, but then I had to travel and lost connectivity. I did expect that Beman was going to look things over and make sure there were no new regressions before the release went out. I hypothesize that part of the problem is that he's not looking at the meta-comm tests, which include Boost.Python and Spirit and show regressions against the previous release, rather than just the last test run. I have been worried for some time that test effectiveness is diluted by having two reporting/display systems... did it bite us here?
No, actually I do look at the meta-comm tests. In fact I review every test on every platform. It takes quite a while. I was also concerned about the Python tests on Linux, and posted a query on January 27th:
Here are the three tests failing gcc 3.3.1 and 3.3.2:
* iterator interoperable_fail
* python embedding
You replied:
This one worries me a little. I'll look into it.
Anyhow, I think your point about multiple reporting is a good one. The volume of tests is just too high. Fewer, more comprehensive, tests would be easier to monitor. Also fewer compilers. Do we really need to test every version of GCC and VC++ for the last four years? If our testing was more focused, we could cycle the tests more often too. --Beman
Beman Dawes wrote:
Anyhow, I think your point about multiple reporting is a good one. The volume of tests is just too high. Fewer, more comprehensive, tests would be easier to monitor.
This could be implemented by using another build target. I'm under the impression that the test results aren't monitored by the library maintainers unless we're close to a release. So, it doesn't make much sense to run all the tests all the time. We had some troubles with process_jam_log recently. If that tool had worked fine then monitoring the test results would have been much simpler.
Also fewer compilers. Do we really need to test every version of GCC and VC++ for the last four years?
One problem is that different Linux distributions prefer different compiler versions and developers in turn seem to prefer to use the compiler shipped by the distribution they use. This results in a great diversity of compiler versions being actively used. Of course, if a decision would be made not to support very old compilers like gcc 2.95 then I'd happily remove them from my list of toolsets.
If our testing was more focused, we could cycle the tests more often too.
We could cycle more often if all test programs could be compiled and run rather fast. However, this is not the case currently. Authors of tests probably are unaware of the compile/ run time their tests need on the systems/compilers they don't use themselves (e.g. random_test takes _very_ long to compile here). Perhaps, it would be helpful to report compile/run times in the test results. Slow tests could be identified easily by displaying the results sorted by the time then. Regards, m
Martin Wille wrote:
Anyhow, I think your point about multiple reporting is a good one. The volume of tests is just too high. Fewer, more comprehensive, tests would be easier to monitor.
This could be implemented by using another build target.
I'm under the impression that the test results aren't monitored by the library maintainers unless we're close to a release. So, it doesn't make much sense to run all the tests all the time.
What about generating an email to the maintainers of the corresponding library, which makes sure they will know about a failed test ... :-) Regards Hartmut
hartmutkaiser@t-online.de (Hartmut Kaiser) writes:
Martin Wille wrote:
Anyhow, I think your point about multiple reporting is a good one. The volume of tests is just too high. Fewer, more comprehensive, tests would be easier to monitor.
This could be implemented by using another build target.
I'm under the impression that the test results aren't monitored by the library maintainers unless we're close to a release. So, it doesn't make much sense to run all the tests all the time.
What about generating an email to the maintainers of the corresponding library, which makes sure they will know about a failed test ... :-)
Yeah, great idea. That's what the BuildBot does. Only, it first sends mail to the person that broke the test, in hopes that (s)he'll fix the problem. That's much better, IMO. -- Dave Abrahams Boost Consulting www.boost-consulting.com
Hartmut Kaiser wrote:
What about generating an email to the maintainers of the corresponding library, which makes sure they will know about a failed test ... :-)
I often had failures due to problems with my local setup. Emails about failed tests should not be sent without being checked by a human before. (Which main- tainer would want to be notified about my disk-full errors?) I reported some problems several times (sometimes to the list sometimes directly to the maintainer). This didn't always result in the problem being fixed. I'm afraid automatically generated emails would be ignored even more. Regards, m
Martin Wille <mw8329@yahoo.com.au> writes:
Hartmut Kaiser wrote:
What about generating an email to the maintainers of the corresponding library, which makes sure they will know about a failed test ... :-)
I often had failures due to problems with my local setup. Emails about failed tests should not be sent without being checked by a human before. (Which main- tainer would want to be notified about my disk-full errors?)
You ;-) Seriously, I'm sure there's a way to deal with this sort of thing, because the BuildBot is being used successfully by several projects.
I reported some problems several times (sometimes to the list sometimes directly to the maintainer). This didn't always result in the problem being fixed. I'm afraid automatically generated emails would be ignored even more.
The way BuildBot works, it first notifies the perpetrator of the offending checkin. After that, if the problem isn't fixed, it can make more-urgent notifications to a different/wider audience. It can use any number of notification methods including email and IRC. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams writes:
Martin Wille <mw8329@yahoo.com.au> writes:
Hartmut Kaiser wrote:
What about generating an email to the maintainers of the corresponding library, which makes sure they will know about a failed test ... :-)
I often had failures due to problems with my local setup.
Same here.
Emails about failed tests should not be sent without being checked by a human before. (Which main- tainer would want to be notified about my disk-full errors?)
You ;-)
Seriously, I'm sure there's a way to deal with this sort of thing,
Probably, but it needs to be figured out, since in our experience misconfiguration/setup issues arise often enough to significantly impact "worthiness" of automatic notifications. Hmm, may be the first step of figuring it out would be to collect and classify such cases. "Disk full" is definitely the first one to be registered (we had it here as well).
becausqe the BuildBot is being used successfully by several projects.
That doesn't necessarily mean that they solved the issue, though. FWIW, it's not mentioned in the paper. -- Aleksey Gurtovoy MetaCommunications Engineering
Aleksey Gurtovoy <agurtovoy@meta-comm.com> writes:
David Abrahams writes:
Martin Wille <mw8329@yahoo.com.au> writes:
Hartmut Kaiser wrote:
What about generating an email to the maintainers of the corresponding library, which makes sure they will know about a failed test ... :-)
I often had failures due to problems with my local setup.
Same here.
Emails about failed tests should not be sent without being checked by a human before. (Which main- tainer would want to be notified about my disk-full errors?)
You ;-)
Seriously, I'm sure there's a way to deal with this sort of thing,
Probably, but it needs to be figured out, since in our experience misconfiguration/setup issues arise often enough to significantly impact "worthiness" of automatic notifications.
Hmm, may be the first step of figuring it out would be to collect and classify such cases. "Disk full" is definitely the first one to be registered (we had it here as well).
becausqe the BuildBot is being used successfully by several projects.
That doesn't necessarily mean that they solved the issue, though. FWIW, it's not mentioned in the paper.
As a matter of fact, Brian emailed me to say that automatic notifications aren't implemented yet: ----
The way BuildBot works, it first notifies the perpetrator of the offending checkin. After that, if the problem isn't fixed, it can make more-urgent notifications to a different/wider audience. It can use any number of notification methods including email and IRC.
FYI, that feature hasn't been implemented yet. There are plans for it (by more than just me), but we haven't yet done the work. At the moment, failing builds are indicated in two ways, both of which require polling: the status web page shows red, and has links to let you see the test failures the optional IRC bot will respond to queries about build status, letting you know that the latest build is "failed" instead of "passed" The "notify the perpetrator" part is easy, as we already have a list of CVS names whose changes are involved in each build. Email and IRC are easy, it's just a question of registering a mapping between CVS names and email addresses / irc nicknames. The "if the problem isn't fixed" part is trickier, as it requires the ability to recognize that two successive test runs have the same failure mode (we need to parse the test case results and identify which test cases are failing). I've found that notification is more effective when the test suite is usually clean. It is easy for a project to get into a mode where there are one or two difficult tests which "always fail", and after a week or two in that state it is hard to attain the motivation to fix them. In that mode, constant "the test suite is still broken" reminders may do more harm than good. The notification scheme needs to take this (human) property into account. In addition, the possibility of intermittent test failures (like the minor timing problems we get on the Twisted tests all the time, usually due to changes in system load) means that we need to be able to recognize them and verify the failure (by re-running the failed test) before harassing anybody. If not, I believe we run the risk of irritating developers to the point of ignoring the buildbot results, which then reduces the point of installing it in the first place. That's why I have (personally) prioritized "push-mode" notification *after* better test-case parsing. cheers, -Brian -- Dave Abrahams Boost Consulting www.boost-consulting.com
Brian:
...I believe we run the risk of irritating developers to the point of ignoring the buildbot results, which then reduces the point of installing
it in the first place.
I'm also quite concerned about irritating developers. One thing I (and perhaps others) need to do is be sure not to put a lot of effort into building a better testing system for broken compiles, when badly broken compilers are becoming a thing of the past. I'll try to give some more though to what that implies for Boost regression test reporting. --Beman
On Thursday 12 February 2004 07:38 am, Martin Wille wrote:
Hartmut Kaiser wrote:
What about generating an email to the maintainers of the corresponding library, which makes sure they will know about a failed test ... :-)
They do this with GCC, and Mark Mitchell has credited it with improved release speed and quality. When something breaks, everyone who has checked in some code since the last successful test run gets a complaining e-mail every day until it gets fixed.
I often had failures due to problems with my local setup. Emails about failed tests should not be sent without being checked by a human before. (Which main- tainer would want to be notified about my disk-full errors?)
Do these failures typically result in a huge number of problems? Perhaps a simple threshold, such as "if there are more than X new failures, e-mail the maintainer of the tests; otherwise, email the people that checked in code last".
I reported some problems several times (sometimes to the list sometimes directly to the maintainer). This didn't always result in the problem being fixed. I'm afraid automatically generated emails would be ignored even more.
The advantage of automatically-generated e-mails is that they would come quickly. For instance, I broken function on GCC 2.95.3 two days ago. I'd looked at a few (but not all) regression results, and hadn't seen any problems, so I forgot about it. Had it not been for David A. telling me directly that I broke something, I would not have known until a release came up and people started looking at regression results more carefully. It's much easier to fix code when you write it the first time, instead of months later in a time crunch. Automatically-generated emails can help that. Doug
Douglas Gregor writes:
On Thursday 12 February 2004 07:38 am, Martin Wille wrote:
I often had failures due to problems with my local setup. Emails about failed tests should not be sent without being checked by a human before. (Which main- tainer would want to be notified about my disk-full errors?)
Do these failures typically result in a huge number of problems? Perhaps a simple threshold, such as "if there are more than X new failures, e-mail the maintainer of the tests; otherwise, email the people that checked in code last".
That might be a good heuristic, although, depending on X, it will either give us some number of false positives (i.e. regression runners will get notified about "real" failures caused by someone's errorneous checkin) or false negatives (number of people will get notified about failures actually caused by a test site problem). False positives would be probably preferable. In any case, this definitely can be a workable solution. -- Aleksey Gurtovoy MetaCommunications Engineering
Martin Wille <mw8329@yahoo.com.au> writes:
If our testing was more focused, we could cycle the tests more often too.
We could cycle more often if all test programs could be compiled and run rather fast. However, this is not the case currently. Authors of tests probably are unaware of the compile/ run time their tests need on the systems/compilers they don't use themselves (e.g. random_test takes _very_ long to compile here).
I agree that egregiously long build times for a test make it prohibitive; those tests need to be fixed. That has been a particular problem with the random library for at least a year and a half. We could also go back to an arrangement in which only outdated tests get run again, so tests that failed last time but for which no dependencies changed wouldn't run again. That said, I don't believe test cycle time is the real problem here. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams writes:
I agree that egregiously long build times for a test make it prohibitive; those tests need to be fixed. That has been a particular problem with the random library for at least a year and a half.
We could also go back to an arrangement in which only outdated tests get run again, so tests that failed last time but for which no dependencies changed wouldn't run again.
Please! Similarly, for "clean" runs, implementing Martin's suggestion would make a _huge_ difference: Martin Wille writes:
2. Not compiling/running tests which are expected to fail. We have a mechanism to mark toolsets unsupported for certain libraries. However, this markup is applied _after_ trying to compile/run the tests. If the build system wouldn't even try to run the tests for unsupported toolsets then this would also speed up a test cycle.
-- Aleksey Gurtovoy MetaCommunications Engineering
At 11:50 PM 2/12/2004, Aleksey Gurtovoy wrote:
David Abrahams writes:
I agree that egregiously long build times for a test make it prohibitive; those tests need to be fixed. That has been a particular problem with the random library for at least a year and a half.
We could also go back to an arrangement in which only outdated tests get run again, so tests that failed last time but for which no dependencies changed wouldn't run again.
Please! Similarly, for "clean" runs, implementing Martin's suggestion would make a _huge_ difference:
Martin Wille writes:
2. Not compiling/running tests which are expected to fail. We have a mechanism to mark toolsets unsupported for certain libraries. However, this markup is applied _after_ trying to compile/run the tests. If the build system wouldn't even try to run the tests for unsupported toolsets then this would also speed up a test cycle.
Yes, that would be a big help. --Beman
On Thu, 2004-02-12 at 11:06, Martin Wille wrote:
I'm under the impression that the test results aren't monitored by the library maintainers unless we're close to a release. So, it doesn't make much sense to run all the tests all the time.
well, personnally I find the batch of test results very useful all the time when I'm working on my library. (and that's not always in the last days before release) I commit, wait a bit, check the regression results of each and every compiler listed, try to understand what seems to be issue with the ones failing, find a workaround, then proceed to next planned commit, etc.. It's true that the tests results are probably watched by less people when far from release, but that doesnt make running all tests all the time useless. hmm, the real problem is each test is run all the time even when no source file used by it has been modified, is that right ? would it be hard to recompile and re-launch only what need be ? -- Samuel
Samuel Krempp wrote:
I commit, wait a bit, check the regression results of each and every compiler listed, try to understand what seems to be issue with the ones failing, find a workaround, then proceed to next planned commit, etc..
I appreciate that. If everyone would work that way then the release procedure wouldn't take as long as it did for 1.31.0.
It's true that the tests results are probably watched by less people when far from release, but that doesnt make running all tests all the time useless.
hmm, the real problem is each test is run all the time even when no source file used by it has been modified, is that right ? would it be hard to recompile and re-launch only what need be ?
It is done that way, actually. However, quite often nearly everything gets recompiled because it needs to be. E.g. every change to Boost.Config or to the type traits causes everything to be recompiled. I don't see any good way to avoid that. Regards, m
One small addition to the regression presentation that would help would be a view organized first by library, and then by OS/compiler. Currently I have to click all over the place to find the results for a single library. Cheers, Jeremy _______________________________________________ Jeremy Siek <jsiek@osl.iu.edu> http://www.osl.iu.edu/~jsiek Ph.D. Student, Indiana University Bloomington C++ Booster (http://www.boost.org) Office phone: (812) 856-1820 _______________________________________________
Jeremy Siek <jsiek@osl.iu.edu> writes:
One small addition to the regression presentation that would help would be a view organized first by library, and then by OS/compiler. Currently I have to click all over the place to find the results for a single library.
Have you looked at the meta-comm summary? Their format is superb for me. -- Dave Abrahams Boost Consulting www.boost-consulting.com
On Feb 12, 2004, at 12:48 PM, David Abrahams wrote:
Have you looked at the meta-comm summary? Their format is superb for me.
Maybe, which page exactly are you referring to? Cheers, Jeremy _______________________________________________ Jeremy Siek <jsiek@osl.iu.edu> http://www.osl.iu.edu/~jsiek Ph.D. Student, Indiana University Bloomington C++ Booster (http://www.boost.org) Office phone: (812) 856-1820 _______________________________________________
Jeremy Siek <jsiek@osl.iu.edu> writes:
On Feb 12, 2004, at 12:48 PM, David Abrahams wrote:
Have you looked at the meta-comm summary? Their format is superb for me.
Maybe, which page exactly are you referring to?
See the developers' summaries at http://www.meta-comm.com/engineering/index.html -- Dave Abrahams Boost Consulting www.boost-consulting.com
Yeah, that is nice, but it only shows results for one OS. On Feb 12, 2004, at 2:58 PM, David Abrahams wrote:
See the developers' summaries at http://www.meta-comm.com/engineering/index.html
-- Dave Abrahams Boost Consulting www.boost-consulting.com
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
_______________________________________________ Jeremy Siek <jsiek@osl.iu.edu> http://www.osl.iu.edu/~jsiek Ph.D. Student, Indiana University Bloomington C++ Booster (http://www.boost.org) Office phone: (812) 856-1820 _______________________________________________
Jeremy Siek <jsiek@osl.iu.edu> writes:
Yeah, that is nice, but it only shows results for one OS.
The Linux page also has meta-comm format links. See the top of http://boost.sourceforge.net/regression-logs/cs-Linux.html IMO, we should be using the meta-comm format exclusively, but apparently there isn't universal agreement on that point. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
Jeremy Siek <jsiek@osl.iu.edu> writes:
Yeah, that is nice, but it only shows results for one OS.
The Linux page also has meta-comm format links. See the top of http://boost.sourceforge.net/regression-logs/cs-Linux.html
IMO, we should be using the meta-comm format exclusively, but apparently there isn't universal agreement on that point.
I'd love to see all the test use that format.. even though I disagree on the L&F of it ;-) Ideally test results would be collected at one location, for example at metacomm, and one set of regression tables generated for everything. This would remove the burden of having to figure out how to set up and run the XML/XSLT programs, increase the likely hood that we can get more people to volunteer hardware time for testing, allow for more complicated analisys of results, etc... -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com - 102708583/icq
Rene Rivera writes:
David Abrahams wrote:
Jeremy Siek <jsiek@osl.iu.edu> writes:
Yeah, that is nice, but it only shows results for one OS. The Linux page also has meta-comm format links. See the top of http://boost.sourceforge.net/regression-logs/cs-Linux.html IMO, we should be using the meta-comm format exclusively, but apparently there isn't universal agreement on that point.
I'd love to see all the test use that format.. even though I disagree on the L&F of it ;-)
Ideally test results would be collected at one location, for example at metacomm, and one set of regression tables generated for everything.
Yep. A single summary report for all platforms would be definitely an improvement over what we have now. We've been planning on it for some time, and it's not hard to implement either, just didn't have enough time to bring it to life.
This would remove the burden of having to figure out how to set up and run the XML/XSLT programs, increase the likely hood that we can get more people to volunteer hardware time for testing, allow for more complicated analisys of results, etc...
All true. -- Aleksey Gurtovoy MetaCommunications Engineering
On Feb 12, 2004, at 3:30 PM, David Abrahams wrote:
Jeremy Siek <jsiek@osl.iu.edu> writes:
Yeah, that is nice, but it only shows results for one OS.
The Linux page also has meta-comm format links. See the top of http://boost.sourceforge.net/regression-logs/cs-Linux.html
And that page also shows results for one OS. My point (albeit small) is that as a developer, at any one time I am not interested in seeing results for all the boost libraries on a given OS. I'm interested in seeing the results for a particular library on all the OS's and compilers. The current organization is probably better for users, but requires a developer to go to many places and do a lot of scanning to see if there are regressions in library X. Another small suggestion that would help a lot would be splitting the compiler error output into separate web pages. Currently those pages are so large it takes a noticeable amount of time to load, which makes it hard to quickly check out several failures. Cheers, Jeremy _______________________________________________ Jeremy Siek <jsiek@osl.iu.edu> http://www.osl.iu.edu/~jsiek Ph.D. Student, Indiana University Bloomington C++ Booster (http://www.boost.org) Office phone: (812) 856-1820 _______________________________________________
At 12:48 PM 2/12/2004, David Abrahams wrote:
Jeremy Siek <jsiek@osl.iu.edu> writes:
One small addition to the regression presentation that would help would be a view organized first by library, and then by OS/compiler. Currently I have to click all over the place to find the results for a single library.
Have you looked at the meta-comm summary? Their format is superb for me.
It only covers the compilers and operating systems they test with. I find myself not looking at it because it doesn't cover a compiler I'm interested in. It would be much more helpful if it covered all compilers and operating systems Boost tests are run on. All of our formats are also painfully slow on less than broadband connections if you try to drill down to see a specific failure. --Beman
Beman Dawes writes:
At 12:48 PM 2/12/2004, David Abrahams wrote:
Jeremy Siek <jsiek@osl.iu.edu> writes:
One small addition to the regression presentation that would help would be a view organized first by library, and then by OS/compiler. Currently I have to click all over the place to find the results for a single library.
Have you looked at the meta-comm summary? Their format is superb for me.
It only covers the compilers and operating systems they test with. I find myself not looking at it because it doesn't cover a compiler I'm interested in.
Out of curiosity, what compiler is it?
It would be much more helpful if it covered all compilers and operating systems Boost tests are run on.
Truly so. It's been on our TODO list for some time. We'll see if we can bring it to life sooner.
All of our formats are also painfully slow on less than broadband connections if you try to drill down to see a specific failure.
They are slow even on broadband. That page definitely needs to be split, somehow. -- Aleksey Gurtovoy MetaCommunications Engineering
At 02:42 AM 2/13/2004, Aleksey Gurtovoy wrote:
Beman Dawes writes:
At 12:48 PM 2/12/2004, David Abrahams wrote:
Jeremy Siek <jsiek@osl.iu.edu> writes:
One small addition to the regression presentation that would help would be a view organized first by library, and then by OS/compiler. Currently I have to click all over the place to find the results for a single library.
Have you looked at the meta-comm summary? Their format is superb for me.
It only covers the compilers and operating systems they test with. I find myself not looking at it because it doesn't cover a compiler I'm interested in.
Out of curiosity, what compiler is it?
GCC and Intel 8.0 are both compilers I watch closely. Incidentally, that isn't a compliant about MetaComm. You guys are doing a great job with both testing and reporting. Thanks! --Beman
Beman Dawes <bdawes@acm.org> writes:
All of our formats are also painfully slow on less than broadband connections if you try to drill down to see a specific failure.
* Summary page is 60K and is OK. * Results page is 800K and might need to be split (or optimized). I think it will be helpful to have to have split and non-split one. * Regression test logs (links.html) file is > 4MB and definetely needs to be split. I've locally patched the reports to split big regression test logs (links.html) file to have one file per test. If this looks OK to everybody I will check the modifications in and inform other regression runners, so they can modify their upload scripts to reflect the changes. -- Misha Bergal MetaCommunications Engineering
On Friday 13 February 2004 04:11 pm, Misha Bergal wrote:
Beman Dawes <bdawes@acm.org> writes: * Regression test logs (links.html) file is > 4MB and definetely needs to be split. I've locally patched the reports to split big regression test logs (links.html) file to have one file per test.
If this looks OK to everybody I will check the modifications in and inform other regression runners, so they can modify their upload scripts to reflect the changes.
Looks great. Doug
At 04:11 PM 2/13/2004, Misha Bergal wrote:
Beman Dawes <bdawes@acm.org> writes:
All of our formats are also painfully slow on less than broadband connections if you try to drill down to see a specific failure.
* Summary page is 60K and is OK.
* Results page is 800K and might need to be split (or optimized). I think it will be helpful to have to have split and non-split one.
* Regression test logs (links.html) file is > 4MB and definetely needs to be split. I've locally patched the reports to split big regression test logs (links.html) file to have one file per test.
If this looks OK to everybody I will check the modifications in and inform other regression runners, so they can modify their upload scripts to reflect the changes.
I'd have to test to see if it is feasible for my ISDN line. My gut feeling is that that many files would have to be zipped up for uploading, and then unzipped on the SourceForge site. We would also have to have a way to clear out old files as tests or compilers were eliminated. OTOH, I do think we need to make changes, so encourage you to experiment. --Beman
At 10:56 AM 2/12/2004, Samuel Krempp wrote:
On Thu, 2004-02-12 at 11:06, Martin Wille wrote:
I'm under the impression that the test results aren't monitored by the library maintainers unless we're close to a release. So, it doesn't make much sense to run all the tests all the time.
well, personnally I find the batch of test results very useful all the time when I'm working on my library. (and that's not always in the last days before release) I commit, wait a bit, check the regression results of each and every compiler listed, try to understand what seems to be issue with the ones failing, find a workaround, then proceed to next planned commit, etc..
That's the development process I'm targeting when I say cycling tests more often would be helpful. The other case is where a developer isn't expecting any change, so isn't even looking at the latest test results. Automatic email notification might be very helpful in that case.
hmm, the real problem is each test is run all the time even when no source file used by it has been modified, is that right ? would it be hard to recompile and re-launch only what need be ?
That's already being done, and with good success. On my setup, with seven compilers, it takes close to three hours to do a run from scratch. If there are just changes to a single library which other libraries don't depend on, it might take 15 minutes. But if nothing at all changed, it still takes, say, 14 minutes. It would be nice to reduce that 14 minutes of "overhead", where some tests are recompiled, even though we know the recompiles will fail, and some tests are re-executed, even though we know the results of re-execution. That being said, we don't want to reduce reliability, of course. --Beman
Samuel Krempp <krempp@crans.ens-cachan.fr> writes:
On Thu, 2004-02-12 at 11:06, Martin Wille wrote:
I'm under the impression that the test results aren't monitored by the library maintainers unless we're close to a release. So, it doesn't make much sense to run all the tests all the time.
well, personnally I find the batch of test results very useful all the time when I'm working on my library. (and that's not always in the last days before release) I commit, wait a bit, check the regression results of each and every compiler listed, try to understand what seems to be issue with the ones failing, find a workaround, then proceed to next planned commit, etc..
Me too.
It's true that the tests results are probably watched by less people when far from release, but that doesnt make running all tests all the time useless.
hmm, the real problem is each test is run all the time even when no source file used by it has been modified, is that right ?
Only failing tests are re-run, unless the tester is paranoid and forces a clean run.
would it be hard to recompile and re-launch only what need be ?
No. -- Dave Abrahams Boost Consulting www.boost-consulting.com
Beman Dawes wrote:
If our testing was more focused, we could cycle the tests more often too.
Of course, there are more ways to support testing at a higher frequency: 1. Having fewer failing tests. This is obvious but also quite relevant. The number of tests failing isn't small. It would help a lot if those would be fixed. 2. Not compiling/running tests which are expected to fail. We have a mechanism to mark toolsets unsupported for certain libraries. However, this markup is applied _after_ trying to compile/run the tests. If the build system wouldn't even try to run the tests for unsupported toolsets then this would also speed up a test cycle. I'd prefer to have the tests fixed over splitting the tests into two or more subsets being run at different occasions. Regards, m
Beman Dawes <bdawes@acm.org> writes:
At 03:51 PM 2/11/2004, David Abrahams wrote:
Well, perhaps we should ask whose responsibility it is to watch the Boost.Python regression logs for VC6.
I doubt there's any assigned responsibility. I was watching, but then I had to travel and lost connectivity. I did expect that Beman was going to look things over and make sure there were no new regressions before the release went out. I hypothesize that part of the problem is that he's not looking at the meta-comm tests, which include Boost.Python and Spirit and show regressions against the previous release, rather than just the last test run. I have been worried for some time that test effectiveness is diluted by having two reporting/display systems... did it bite us here?
No, actually I do look at the meta-comm tests. In fact I review every test on every platform. It takes quite a while. I was also concerned about the Python tests on Linux, and posted a query on January 27th:
Here are the three tests failing gcc 3.3.1 and 3.3.2:
* iterator interoperable_fail
That one was expected.
* python embedding
You replied:
This one worries me a little. I'll look into it.
And IIRC I fixed that problem. Different tests that are apparently failing in 1.31.0.
Anyhow, I think your point about multiple reporting is a good one. The volume of tests is just too high. Fewer, more comprehensive, tests would be easier to monitor. Also fewer compilers. Do we really need to test every version of GCC and VC++ for the last four years?
Yes, IMO, if people want to support those compilers, we do need to test them.
If our testing was more focused, we could cycle the tests more often too.
I'm really unconvinced that we need smaller tests on fewer compilers. Human monitoring is just too error-prone. Why risk it? Why not have comprehensive tests with automated notifications when something breaks? It seems to me that less testing can only result in fewer things working, and coupled with human monitoring it's just going to make things worse, not better. -- Dave Abrahams Boost Consulting www.boost-consulting.com
On Feb 12, 2004, at 8:30 AM, David Abrahams wrote:
I'm really unconvinced that we need smaller tests on fewer compilers. Human monitoring is just too error-prone. Why risk it? Why not have comprehensive tests with automated notifications when something breaks? It seems to me that less testing can only result in fewer things working, and coupled with human monitoring it's just going to make things worse, not better.
This sounds like the right direction to go in. Cheers, Jeremy _______________________________________________ Jeremy Siek <jsiek@osl.iu.edu> http://www.osl.iu.edu/~jsiek Ph.D. Student, Indiana University Bloomington C++ Booster (http://www.boost.org) Office phone: (812) 856-1820 _______________________________________________
At 08:30 AM 2/12/2004, David Abrahams wrote:
Beman Dawes <bdawes@acm.org> writes:
Anyhow, I think your point about multiple reporting is a good one. The volume of tests is just too high. Fewer, more comprehensive, tests would be easier to monitor. Also fewer compilers. Do we really need to test every version of GCC and VC++ for the last four years?
Yes, IMO, if people want to support those compilers, we do need to test them.
It comes down to resources. I suppose if people really care about older compilers, they will be willing to contribute testing.
If our testing was more focused, we could cycle the tests more often too.
I'm really unconvinced that we need smaller tests on fewer compilers.
I don't think smaller tests is a good idea. I'm asking for "more comprehensive" tests. Say a library now covers 100 test cases spread out among five test programs. I'd like to see those five programs refactored into one program, still covering the 100 test cases. Or even adding more test cases. A single program would cut the overhead, including the human overhead. It would be easier to know if any particular tests was a resource problem if there was a bjam option to time the build and run aspects of the tests. The change Jeremy made recently to one or two of the graph tests noticeably sped testing, but it was only happenstance that I realized they were problems.
Human monitoring is just too error-prone. Why risk it? Why not have comprehensive tests with automated notifications when something breaks? It seems to me that less testing can only result in fewer things working, and coupled with human monitoring it's just going to make things worse, not better.
Great! Let's give it a try. Who has to do what to start? --Beman
Beman Dawes wrote:
I don't think smaller tests is a good idea. I'm asking for "more comprehensive" tests.
Say a library now covers 100 test cases spread out among five test programs. I'd like to see those five programs refactored into one program, still covering the 100 test cases. Or even adding more test cases. A single program would cut the overhead, including the human overhead.
There are at least three drawbacks to this approach: 1. "something is wrong" is all the information you get from a failing test. Esp. you'll likely see only one of several problems related to a failing test program. The next problem will only become visible after the first problem has been fixed. 2. Some tests are known to fail for certain compilers. If those tests are joined with other tests then we'll lose information about these other tests. 3. Compile time may become very large for large test programs or heavy template usage. E.g. in one case, we had to split a test into three (Spirit's switch_p tests) in order to make testing feasible. Regards, m
Martin Wille <mw8329@yahoo.com.au> writes:
Beman Dawes wrote:
I don't think smaller tests is a good idea. I'm asking for "more comprehensive" tests. Say a library now covers 100 test cases spread out among five test programs. I'd like to see those five programs refactored into one program, still covering the 100 test cases. Or even adding more test cases. A single program would cut the overhead, including the human overhead.
There are at least three drawbacks to this approach:
1. "something is wrong" is all the information you get from a failing test. Esp. you'll likely see only one of several problems related to a failing test program. The next problem will only become visible after the first problem has been fixed.
2. Some tests are known to fail for certain compilers. If those tests are joined with other tests then we'll lose information about these other tests.
3. Compile time may become very large for large test programs or heavy template usage. E.g. in one case, we had to split a test into three (Spirit's switch_p tests) in order to make testing feasible.
I agree. We already have another example of such a problem in the random library test. What contortions do we have to do in order to get that test refactored? How hard can it be? -- Dave Abrahams Boost Consulting www.boost-consulting.com
At 12:35 PM 2/12/2004, Martin Wille wrote:
Beman Dawes wrote:
I don't think smaller tests is a good idea. I'm asking for "more comprehensive" tests.
Say a library now covers 100 test cases spread out among five test programs. I'd like to see those five programs refactored into one program, still covering the 100 test cases. Or even adding more test cases. A single program would cut the overhead, including the human overhead.
There are at least three drawbacks to this approach:
1. "something is wrong" is all the information you get from a failing test. Esp. you'll likely see only one of several problems related to a failing test program. The next problem will only become visible after the first problem has been fixed.
That's only correct for new, immature code. Many Boost libraries are now mature. They pass all tests, except in exceptional circumstances.
2. Some tests are known to fail for certain compilers. If those tests are joined with other tests then we'll lose information about these other tests.
Most compilers are now passing close to 100% of all tests. Hopefully, with the next round of compiler updates, they will be passing every test. Granularity on 100% passes brings no benefits.
3. Compile time may become very large for large test programs or heavy template usage. E.g. in one case, we had to split a test into three (Spirit's switch_p tests) in order to make testing feasible.
It is hard to know the overall effect without accurate timings. My personal belief is that on average the total time will drop. But we need timings to know for sure. --Beman
Beman Dawes <bdawes@acm.org> writes:
3. Compile time may become very large for large test programs or heavy template usage. E.g. in one case, we had to split a test into three (Spirit's switch_p tests) in order to make testing feasible.
It is hard to know the overall effect without accurate timings. My personal belief is that on average the total time will drop. But we need timings to know for sure.
I still can't understand why we're focused on reducing testing time. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
Beman Dawes <bdawes@acm.org> writes:
3. Compile time may become very large for large test programs or heavy template usage. E.g. in one case, we had to split a test into three (Spirit's switch_p tests) in order to make testing feasible.
It is hard to know the overall effect without accurate timings. My personal belief is that on average the total time will drop. But we need timings to know for sure.
I still can't understand why we're focused on reducing testing time.
I'd appreciate shorter testing times very much. A clean run for 1.31.0 took more than half a day here. (Having only 256MB RAM slows the compilation process down rather drastically. The actual execution time for the test programs has been a problem only on very rare occasions.) Regards, m
At 11:18 PM 2/12/2004, David Abrahams wrote:
Beman Dawes <bdawes@acm.org> writes:
3. Compile time may become very large for large test programs or heavy template usage. E.g. in one case, we had to split a test into three (Spirit's switch_p tests) in order to make testing feasible.
It is hard to know the overall effect without accurate timings. My personal belief is that on average the total time will drop. But we need timings to know for sure.
I still can't understand why we're focused on reducing testing time.
The longer the tests take, the fewer times they can be cycled. That is a particular problem as a release nears. The release manager often has to delay some action pending the outcome of tests. Not a problem if the tests run quickly and cycle often. But when a test cycle takes 2 1/2 to 3 hours, as was common during the runup for 1.31.0, it adds days to the time it takes to finish a release. --Beman
Beman Dawes <bdawes@acm.org> writes:
At 11:18 PM 2/12/2004, David Abrahams wrote:
Beman Dawes <bdawes@acm.org> writes:
3. Compile time may become very large for large test programs or heavy template usage. E.g. in one case, we had to split a test into three (Spirit's switch_p tests) in order to make testing feasible.
It is hard to know the overall effect without accurate timings. My personal belief is that on average the total time will drop. But we need timings to know for sure.
I still can't understand why we're focused on reducing testing time.
The longer the tests take, the fewer times they can be cycled.
That is a particular problem as a release nears. The release manager often has to delay some action pending the outcome of tests. Not a problem if the tests run quickly and cycle often. But when a test cycle takes 2 1/2 to 3 hours, as was common during the runup for 1.31.0, it adds days to the time it takes to finish a release.
But if we were only running the outdated tests, this would be much less of an issue, right? -- Dave Abrahams Boost Consulting www.boost-consulting.com
At 09:26 AM 2/16/2004, David Abrahams wrote:
Beman Dawes <bdawes@acm.org> writes:
At 11:18 PM 2/12/2004, David Abrahams wrote:
Beman Dawes <bdawes@acm.org> writes:
3. Compile time may become very large for large test programs or heavy template usage. E.g. in one case, we had to split a test into three (Spirit's switch_p tests) in order to make testing feasible.
It is hard to know the overall effect without accurate timings. My personal belief is that on average the total time will drop. But we need timings to know for sure.
I still can't understand why we're focused on reducing testing time.
The longer the tests take, the fewer times they can be cycled.
That is a particular problem as a release nears. The release manager often has to delay some action pending the outcome of tests. Not a problem if the tests run quickly and cycle often. But when a test cycle takes 2 1/2 to 3 hours, as was common during the runup for 1.31.0, it adds days to the time it takes to finish a release.
But if we were only running the outdated tests, this would be much less of an issue, right?
Yes. Also note that we've talked about other changes which also will serve to minimize compile and/or test times. Dropping outdated compilers. Reworking tests which take excessive times. Etc. The combined effect could be markedly reduced regression testing load. --Beman
Beman Dawes <bdawes@acm.org> writes:
At 12:35 PM 2/12/2004, Martin Wille wrote: [snip]
3. Compile time may become very large for large test programs or heavy template usage. E.g. in one case, we had to split a test into three (Spirit's switch_p tests) in order to make testing feasible.
It is hard to know the overall effect without accurate timings. My personal belief is that on average the total time will drop. But we need timings to know for sure.
There are some cases when really big tests would certainly be slower. I had to split some of my Python container library tests into separate modules because g++ was starting to thrash the swapfile(s) on my home machine (with 256MB of RAM). I would guess that otherwise one big test module would certainly be faster, but there is a practical limit there somewhere. -- Raoul Gough. export LESS='-X'
At 03:31 AM 2/13/2004, Raoul Gough wrote:
Beman Dawes <bdawes@acm.org> writes:
At 12:35 PM 2/12/2004, Martin Wille wrote: [snip]
3. Compile time may become very large for large test programs or heavy template usage. E.g. in one case, we had to split a test into three (Spirit's switch_p tests) in order to make testing feasible.
It is hard to know the overall effect without accurate timings. My personal belief is that on average the total time will drop. But we need timings to know for sure.
There are some cases when really big tests would certainly be slower. I had to split some of my Python container library tests into separate modules because g++ was starting to thrash the swapfile(s) on my home machine (with 256MB of RAM). I would guess that otherwise one big test module would certainly be faster, but there is a practical limit there somewhere.
Yes. GCC is a particular problem because of its tendency to run away with RAM. I even have trouble sometimes with a 1 Gigabyte system. --Beman
Beman Dawes <bdawes@acm.org> wrote:
Most compilers are now passing close to 100% of all tests. Hopefully, with the next round of compiler updates, they will be passing every test. Granularity on 100% passes brings no benefits
That's still at least one year away from now, I think. And we need to think when to stop supporting MSVC6, GCC2.95, BCB6 etc. old compilers. As long as these compilers are supported, this granularity is usefull. Actually, I'm little surprised that this thread has not come (yet) to the issue of supporting old compilers. IMVHO, when it comes to testing, these compilers are real problem. B.
Bronek Kozicki wrote:
Beman Dawes <bdawes@acm.org> wrote:
Most compilers are now passing close to 100% of all tests. Hopefully, with the next round of compiler updates, they will be passing every test. Granularity on 100% passes brings no benefits
That's still at least one year away from now, I think. And we need to think when to stop supporting MSVC6, GCC2.95, BCB6 etc. old compilers. As long as these compilers are supported, this granularity is usefull.
I only speak for myself, but initially trying 1.31 in my fairly large project with BCB6 is crashing. It looks as though I'm getting a dodgy result from a boost::function, which I haven't looked at carefully, but I think is due to compiler code generation bugs. I do have other places where I get code gen bugs. I don't have the time to investigate this, so I'm going to stick with 1.30.2 for the foreseeable future until I can upgrade to a different compiler (spirit in 1.31 already doesn't support bcc32 5.6) so what I'm trying to say is I wouldn't complain if future boost releases didn't support bcc32 anymore but I may well be alone in this opinion. I've also got problems that I can no longer put file system headers into pre-compiled headers due to internal compiler errors and other little niggles. All I think due to compiler bugs, but as I said, don't really have the time to investigate now so am happy to stick with 1.30.2. As spirit has done, you do have to stop support for non-conforming compilers at some point, and 1.30.2 and 1.31 are very functional versions of boost, so maybe a good point to stop having to put all the work-arounds in. Cheers Russell
At Friday 2004-02-13 03:26, you wrote:
Beman Dawes <bdawes@acm.org> wrote:
Most compilers are now passing close to 100% of all tests. Hopefully, with the next round of compiler updates, they will be passing every test. Granularity on 100% passes brings no benefits
That's still at least one year away from now, I think. And we need to think when to stop supporting MSVC6, GCC2.95, BCB6 etc. old compilers. As long as these compilers are supported, this granularity is usefull.
Actually, I'm little surprised that this thread has not come (yet) to the issue of supporting old compilers. IMVHO, when it comes to testing, these compilers are real problem.
when it comes to writing/updating the libraries, support of these fossils is staggeringly expensive.
B.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Victor A. Wagner Jr. http://rudbek.com The five most dangerous words in the English language: "There oughta be a law"
"Victor A. Wagner, Jr." <vawjr@rudbek.com> writes:
Actually, I'm little surprised that this thread has not come (yet) to the issue of supporting old compilers. IMVHO, when it comes to testing, these compilers are real problem.
when it comes to writing/updating the libraries, support of these fossils is staggeringly expensive.
I just want to point out that the "fossils" with major conformance problems are still in extremely wide use in industry, and some are even the latest official releases (see Borland). One of my most progressive and forward-thinking clients is still using vc6 for production work. They're anxious to upgrade to vc7.1, but haven't yet had the opportunity. I'd love to stop hacking around vc6 and borland limitations, but we should be sure we understand what we're doing when that day comes. -- Dave Abrahams Boost Consulting www.boost-consulting.com
On Mon, 16 Feb 2004 09:31:25 -0500, David Abrahams wrote
"Victor A. Wagner, Jr." <vawjr@rudbek.com> writes:
when it comes to writing/updating the libraries, support of these fossils is staggeringly expensive.
Agree...
I just want to point out that the "fossils" with major conformance problems are still in extremely wide use in industry, and some are
I agree with Dave on this point. The issue for projects, especially with a large working source base deployed to the world, is that the time to upgrade isn't obviously worth the cost and risk of upgrading. It's usually not the development effort that is the problem, it's the testing costs.
even the latest official releases (see Borland). One of my most progressive and forward-thinking clients is still using vc6 for production work. They're anxious to upgrade to vc7.1, but haven't yet had the opportunity. I'd love to stop hacking around vc6 and borland limitations, but we should be sure we understand what we're doing when that day comes.
I think that the policy Spirit has adopted and I am using for date-time are reasonable going forward. Basically, the concept is to keep the current level of support for the library. Thus if a new feature is added to the library and it just works on the legacy compiler that's fine. If there are problems then that compiler that is noted in the compiler support information and that's it. This level of support still requires work to avoid breaking features that are currently working, but not as much. A year from now, I think we could go to the next step and drop support altogether for these older compilers -- meaning stop regression testing against the old compilers. Jeff
At Monday 2004-02-16 07:31, you wrote:
"Victor A. Wagner, Jr." <vawjr@rudbek.com> writes:
Actually, I'm little surprised that this thread has not come (yet) to the issue of supporting old compilers. IMVHO, when it comes to testing, these compilers are real problem.
when it comes to writing/updating the libraries, support of these fossils is staggeringly expensive.
I just want to point out that the "fossils" with major conformance problems are still in extremely wide use in industry, and some are even the latest official releases (see Borland). One of my most progressive and forward-thinking clients is still using vc6 for production work. They're anxious to upgrade to vc7.1, but haven't yet had the opportunity. I'd love to stop hacking around vc6 and borland limitations, but we should be sure we understand what we're doing when that day comes.
I understand the pain of updating, and I understand even more the pain of working with stuff that is flat out broken. This will no doubt sound very arrogant to many, but IMO, we've coddled Microsoft for WAY to long. They almost killed C++ by their steadfast refusal for many years to NOT adhere to the standard, and the only way we could have pressured them would have been to drop support for their broken compiler (VC6.0 IS broken, make NO mistake about it). How many (wo)man years have been wasted by the human race making workarounds for a compiler that was broken shortly after it was released? Everyone seems to forget that Microsoft publicly announced that the wouldn't make their C++ ISO conformant because "Our customers are more interested in backwards compatibility than in standards conformance." How many more (wo)man years will we waste by telling people "it's ok, you don't have to upgrade, we'll do all the hard work for your broken compiler" ? Is this how we make progress? I mean, it's _only_ been 5 years since the standard was approved. If you'd love to stop, then stop. "they haven't had the opportunity".... and as long as you keep doing the workarounds for them they don't have a compelling reason to upgrade. I certainly hope you get paid well for the pain. Support doesn't just impact the users of these fossils, it affects all of us. We have to wait longer for things to be released. We have to slow down our compiles doing all the #ifdef BROKEN_COMPILER_3 #endif stuff in the headers We have difficulty reading the source because of all the workarounds tucked in. Workarounds don't necessarily get removed when the compiler is "fixed". I see messages from VC6 users that are having problems w/ parts of boost... it takes my time to skip over them in the EMail, it takes bandwidth and disk space. It's been, what?, 2 years now since Microsoft replaced VC6.0? How long are we going to be held back by a company that _clearly_ wanted to bury C++ 7 years ago? In some sense we seem to be suffering from the same thing that battered spouses suffer. We're afraid to leave, even when there is clearly a better alternative. I note with approval that Spirit 1.8 has dropped support for "substantially non-conforming" compilers. I believe we know the consequences "when the day comes". I think the day went some time ago.
-- Dave Abrahams Boost Consulting www.boost-consulting.com
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Victor A. Wagner Jr. http://rudbek.com The five most dangerous words in the English language: "There oughta be a law"
On Mon, 16 Feb 2004 23:57:57 -0700, Victor A. Wagner, Jr. wrote:
How many more (wo)man years will we waste by telling people "it's ok, you don't have to upgrade, we'll do all the hard work for your broken compiler" ?
they still will need to upgrade from MSVC6 before its end-of-life, which is announced at the end of September this year (or pay for the support one more year). Of course, everyone (having bought license) may use MSVC6 after this time, but I see no point why we (or anybody else) would want to fully support it at this time, when it's no longer supported by producer.
Is this how we make progress? I mean, it's _only_ been 5 years since the standard was approved.
and MSCV6 was released before standard, in August 1998. I'm really not sure that introducing serious feature changes (to comply with standard) in all these service packs (which came later) would be good thing to do. To make the point - MSVC6 is *really* old compiler, it was released before standard and soon won't be supported any more (unless you want to spend extra money or just do not care about support).
I note with approval that Spirit 1.8 has dropped support for "substantially non-conforming" compilers.
I like it too. And I also like that there is still support for version 1.6.1. . Maybe we should consider following this path, ie. create new releases without all these nasty workarounds, while still supporting older release of boost when used with old compilers ? B. PS. some links about Microsoft developer tool lifetime: http://support.microsoft.com/default.aspx?pr=lifeDevTool http://support.microsoft.com/default.aspx?scid=fh;[ln];LifeDevToolFam
Bronek Kozicki wrote:
I like it too. And I also like that there is still support for version 1.6.1. . Maybe we should consider following this path, ie. create new releases without all these nasty workarounds, while still supporting older release of boost when used with old compilers ?
I was thinking the same, that maybe patches could still be applied to the 1.30.2 (or whatever version support for older compilers is dropped) branch (on a much lesser priority than for 1.31.0) so at some point, a bug fix release could be shipped. Cheers Russell
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Victor A. Wagner, Jr. Sent: 17 February 2004 06:58 To: Boost mailing list Subject: Re: [boost] Re: 1.31.0 regressions I won't waste bandwidth by getting into rant mode, rut I agree that the time has come to no longer encourage developers to support the obselete compilers. If the code works on VC 6, fine: if not, tough. As Victor A. Wagner rightly observes, we all pay something for continuing support. Those still using the obselete must balance the value of the newer Boost items with the upgrade costs. Paul Paul A Bristow Prizet Farmhouse, Kendal, Cumbria UK LA8 8AB +44 1539 561830 +44 7714 330204 mailto: pbristow@hetp.u-net.com
"Boost" <boost@hetp.u-net.com> writes:
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Victor A. Wagner, Jr. Sent: 17 February 2004 06:58 To: Boost mailing list Subject: Re: [boost] Re: 1.31.0 regressions
I won't waste bandwidth by getting into rant mode, rut I agree that the time has come to no longer encourage developers to support the obselete compilers.
AFAIK, there is no Boost policy of encouraging support for old compilers (they're not obsolete yet, unfortunately). Each developer makes his own choices in response to user demand.
If the code works on VC 6, fine: if not, tough. As Victor A. Wagner rightly observes, we all pay something for continuing support.
Those still using the obselete must balance the value of the newer Boost items with the upgrade costs.
I expect that dropping vc6 support in Boost.Python in the next couple of years would significantly reduce its audience, so I can't justify it, (maybe not until Python's standard Windows distribution drops VC6 also). That fact has caused me to implement vc6 support in other libraries that Boost.Python depends on. Thereafter, if the library breaks, it's a regression. It's viral :( -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
I expect that dropping vc6 support in Boost.Python in the next couple of years would significantly reduce its audience, so I can't justify it, (maybe not until Python's standard Windows distribution drops VC6 also). That fact has caused me to implement vc6 support in other libraries that Boost.Python depends on. Thereafter, if the library breaks, it's a regression. It's viral :(
Yes, but we have a very functional set of libraries in boost-1.30.2 and boost-1.31.0 so why not say older compilers are stuck to using these versions of boost? We're not saying they can't use boost, just not the latest version? Russell
On Tue, 17 Feb 2004 18:32:51 +0000 Russell Hind <rhind@mac.com> wrote:
Yes, but we have a very functional set of libraries in boost-1.30.2 and boost-1.31.0 so why not say older compilers are stuck to using these versions of boost? We're not saying they can't use boost, just not the latest version?
I agree. Some very nice libraries (e.g., ACE) have, IMO, been significantly hindered because of the vast array of compiler/OS versions they support (this support is a main reason for its growth and acceptance in the first place). At some point, the individual boost libraries will have to kick the parasites to the curb... or continued development will be significantly impaired, not to mention it will be impossible to keep sanity in the release-master role. The release master is saddled with making sure all these versions still work, which is a terrible burden, and vastly increases the time for testing, and releasing subsequent versions of the code base.
Russell Hind <rhind@mac.com> writes:
David Abrahams wrote:
I expect that dropping vc6 support in Boost.Python in the next couple of years would significantly reduce its audience, so I can't justify it, (maybe not until Python's standard Windows distribution drops VC6 also). That fact has caused me to implement vc6 support in other libraries that Boost.Python depends on. Thereafter, if the library breaks, it's a regression. It's viral :(
Yes, but we have a very functional set of libraries in boost-1.30.2 and boost-1.31.0 so why not say older compilers are stuck to using these versions of boost? We're not saying they can't use boost, just not the latest version?
Might be a good idea. I think I'd like to continue to support vc6 at least on the 1.31.x branch, particularly because last-minute changes in the graph library broke Boost.Python on vc6 for 1.31.0 What would we do about Borland, which is in some ways more broken than vc6? They don't ship a compiler I wouldn't consider broken. What about GCC 2.9Xes, which are standard equipment on some widely-used Linux distros? Way less broken than either of those two compilers, but still way out-of-date... -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
Russell Hind <rhind@mac.com> writes:
[...]
Yes, but we have a very functional set of libraries in boost-1.30.2 and boost-1.31.0 so why not say older compilers are stuck to using these versions of boost? We're not saying they can't use boost, just not the latest version?
Might be a good idea. I think I'd like to continue to support vc6 at least on the 1.31.x branch, particularly because last-minute changes in the graph library broke Boost.Python on vc6 for 1.31.0
What would we do about Borland, which is in some ways more broken than vc6? They don't ship a compiler I wouldn't consider broken. What about GCC 2.9Xes, which are standard equipment on some widely-used Linux distros? Way less broken than either of those two compilers, but still way out-of-date...
I think most distros finally switched to gcc 3. I also think dropping gcc 2.9x support would hit fewer developers than dropping VC6 support (mostly due to the fact that gcc 3 has been around for some time). However, I didn't conduct a survey. I think it would be reasonable not to support gcc 2.9x in future Boost releases. Regards, m
David Abrahams wrote:
Might be a good idea. I think I'd like to continue to support vc6 at least on the 1.31.x branch, particularly because last-minute changes in the graph library broke Boost.Python on vc6 for 1.31.0
What would we do about Borland, which is in some ways more broken than vc6? They don't ship a compiler I wouldn't consider broken.
Well, I only use bcc32 and for now, I'm happy to stick with 1.30.2. I'll back port any new features I need myself (mainly from filesystem/thread if any attractive features come in) and I may move to 1.31.0 but I've run into other problems using this at the moment so I can only speak for myself about bcc32, but as spirit has already dropped support for bcc32 in 1.31.0 (spirit-1.8 IIRC) then it means boost can't be used 'out-of-the-box' anyway, and requires still having 1.30.2 around anyway. But the new compiler is in preview (bccx 6.0) and already works with quite a bit of boost (thanks to John Maddock) and there hasn't been a word yet, but it has been in preview for about 5 months now and the talk was of the next version of CBX (hopefully with the final product int) due 2-3 quarter this year, so we may be able to get Borland support back in the not too distant future (without all the work-arounds currently needed). Thanks Russell
On Tue, 17 Feb 2004 16:03:11 -0500, David Abrahams wrote:
Might be a good idea. I think I'd like to continue to support vc6 at least on the 1.31.x branch, particularly because last-minute changes in the graph library broke Boost.Python on vc6 for 1.31.0
If we agree to move in this direction, then in release 1.32 (or little later) we will be able to drop many workarounds from boost and simplify testing. But there is question that you raised:
What would we do about Borland, which is in some ways more broken than vc6? They don't ship a compiler I wouldn't consider broken.
I heard some voices that their new compiler being currently in beta (or maybe release candidate?) will be really close to standard. But we can be sure that old Borland compilers will still be widely in use, partialy due to lack of VCL support in this new compiler.
What about GCC 2.9Xes, which are standard equipment on some widely-used Linux distros? Way less broken than either of those two compilers, but still way out-of-date...
I would think that users of MSVC6, old Borland compilers or GCC 2.95 are "adoption blockers" and they possibly do not care much about new features in boost (apparently they also do not care much about decent implementation of C++ standard library). Thus hopefully they will not mind if we just ask them to use releases in 1.31.x branch. But it also means that we will need to support two branches for (at least) next two years. This might be cumbersome, but on the other hand it should simplify development in "current" branch. B.
Bronek Kozicki wrote:
I heard some voices that their new compiler being currently in beta (or maybe release candidate?) will be really close to standard. But we can be sure that old Borland compilers will still be widely in use, partialy due to lack of VCL support in this new compiler.
Yes, the new compiler is much better but is in beta (no mult-threaded support / mt rtl yet etc) and no word as to whether it will support the required extensions to build VCL applications. Borland haven't announced there plans for VCL yet so this isn't a definite no. OTOH, Borland have basically dropped support for bcc32 with the last patch of BCB6 (probably 2 years ago now?) so why should boost continue to support it? It isn't going to get any better. bccx will hopefuly be released in the first half of this year, so I would look to supporting that rather than bcc32 from now on. We have to accept that Borland have dropped support for the old compiler and yes, if Borland don't add VCL support to the new compiler, then a lot of people (me included) won't be able to migrate to the new compiler in a hurry, but that is Borland's problem and as customers we have to let Borland now what we need, but boost should just keep moving forward in the best possible manner, and bccx is Borland's future compiler, and will provide much less headaches for the maintainers in terms of work-arounds etc. Cheers Russell
From: "Russell Hind" <rhind@mac.com>
OTOH, Borland have basically dropped support for bcc32 with the last patch of BCB6 (probably 2 years ago now?) so why should boost continue to support it?
1. People have vast quantities of legacy BCB code. They may like to improve their code with Boost. 2. BCB is still (IMHO) best C++ tool for GUI applications, despite its bugginess. 3. It is far from clear if there will be new Borland C++ compiler at all. Borland may decide to drop out of C++ market completely - it may be waste to spend effort with beta until things get clear. IMO there should be no discouragement against BCB and if it is possible to support it reasonably, it should be done. /Pavel
Pavel Vozenilek wrote:
1. People have vast quantities of legacy BCB code. They may like to improve their code with Boost.
I am in that position, but I am happy to accept that while I'm using effectively a compiler that Borland no longer support, then I stick with the tools I have (much as we don't know if there will be another version of the VCL, so we are stuck with VCL 6) I'm happy to accept that I have to stick with the current version of boost. I know what its capabilities are. If a new feature comes into boost 1.x then why not attempt to back port it to 1.31.x and get a new roll out of 1.31.x for that bcc32. I will consider this as and when new features are added to the thread library as I'm using 1.30.2 and plan to stick with it for the foreseeable future.
2. BCB is still (IMHO) best C++ tool for GUI applications, despite its bugginess.
I agree, but I'm still confident that CBX 2 will come out and at some point we'll have a full bccx and full version of wxWindows and their GUI designer. I've also accepted that it may mean a re-work of the GUI code to wxWindows from VCL to use the new IDE but that is now/less harder than a re-work to .Net or another platform. There isn't a lot I can do about it. I stick with what I have now (boost-1.30.2, VCL 6, bcc32 5.6.4) or move to another platform. If the VCL and compiler aren't going to be updated, then why should boost developers get a headache trying to support this compiler with new features? And if we're lucky, Borland may update the VCL and the new compiler to support VCL extensions and in that case great, we'll get boost to work with the new compiler (which from initial testing hardly needs any workarounds so little problems for the boost developers).
3. It is far from clear if there will be new Borland C++ compiler at all. Borland may decide to drop out of C++ market completely - it may be waste to spend effort with beta until things get clear.
In that case again, why should boost keep supporting it if Borland won't? We have a very good C++ development tool in BCB6 with VCL and boost-1.31.0 or 1.30.2. But the compiler has limitations and we can't expect it to work for ever. Cheers Russell
Russell Hind wrote:
Pavel Vozenilek wrote:
[...]
3. It is far from clear if there will be new Borland C++ compiler at all. Borland may decide to drop out of C++ market completely - it may be waste to spend effort with beta until things get clear.
In that case again, why should boost keep supporting it if Borland won't?
Because I may want to? Why should I be prevented from supporting bcc if I so desire? Should I be forced to run my own regression tests in order to provide that support?
Peter Dimov <pdimov@mmltd.net> wrote:
Because I may want to? Why should I be prevented from supporting bcc if I so desire? Should I be forced to run my own regression tests in order to provide that support?
I think nobody is going to prevent you from supporting any compiler you want :> However, I do suggest that *if there is new branch* of boost, many regression tests targeted at old (nonconforming) compilers should be pulled out from this branch. Effectively, to make new features available for old compilers, these features have to be back-ported to the "old branch". It's just an idea how this could look like - by no means I'm suggesting it's best way to go. B.
Bronek Kozicki wrote:
Peter Dimov <pdimov@mmltd.net> wrote:
Because I may want to? Why should I be prevented from supporting bcc if I so desire? Should I be forced to run my own regression tests in order to provide that support?
I think nobody is going to prevent you from supporting any compiler you want :> However, I do suggest that *if there is new branch* of boost, many regression tests targeted at old (nonconforming) compilers should be pulled out from this branch.
That's what I disagree with. I want these compilers to participate in the regression tests, so that I can see when I inadvertently introduce a regression.
On Wed, 18 Feb 2004 16:13:20 +0200, Peter Dimov wrote
Bronek Kozicki wrote:
Peter Dimov <pdimov@mmltd.net> wrote:
Because I may want to? Why should I be prevented from supporting bcc if I so desire? Should I be forced to run my own regression tests in order to provide that support?
I think nobody is going to prevent you from supporting any compiler you want :> However, I do suggest that *if there is new branch* of boost, many regression tests targeted at old (nonconforming) compilers should be pulled out from this branch.
That's what I disagree with. I want these compilers to participate in the regression tests, so that I can see when I inadvertently introduce a regression.
I agree, I don't want the old compilers removed from the regression test sets. Because, my goal for date_time is to keep the current level of support in the old compilers. Thus if a new feature is added to the library and it just works on the legacy compiler that's fine. If there are problems then that compiler that is noted as failing in the compiler support information and that's it. This level of support still requires work to avoid breaking features that are currently working, but not as much as full support. A year from now, I think we could go to the next step and drop support altogether for these older compilers -- meaning stop regression testing against the old compilers. Jeff
From: "Bronek Kozicki" <brok@rubikon.pl>
However, I do suggest that *if there is new branch* of boost, many regression tests targeted at old (nonconforming) compilers should be pulled out from this branch. Effectively, to make new features available for old compilers, these features have to be back-ported to the "old branch".
I am afraid it woudn't work very well: - There are many small fixed and updates and these could be easily be missed. - The effort to keep multiple branches of Boost would take more than maintaining current state. I'd prefere to keep current mode for 1-2 years. /Pavel
Because I may want to? Why should I be prevented from supporting bcc if I so desire? Should I be forced to run my own regression tests in order to provide that support?
And if you want to then do so. I'm not asking for boost to explicitly drop bcc32 as a supported compiler, just saying that I'm happy with 1.30 and 1.31 with bcc32 and realise the compiler has limitations, so if boost developers were to say it is too much effort supporting the old compiler then I'm quite happy for that and to stick with what I have so I know where I stand. I know which features work and which don't. You are free to support what compilers you like. If I were to write a library and submit to boost, then I would support bcc32 because that is my compiler of choice but I'm going to try and 'force' the boost developers to support compiler and hinder development on the library because of that. Thanks Russell
Peter Dimov wrote:
Because I may want to? Why should I be prevented from supporting bcc if I so desire? Should I be forced to run my own regression tests in order to provide that support?
But to look at it another way: You may want to support bcc32 in your library, but someone else may not want to support it in another library. (e.g. spirit) for the next version of boost I use both your library and spirit from boost. In the next version of boost, spirit no longer supports bcc32 (as in 1.31.x) so I'm sticking with 1.30.2 because I suddenly not all my code works with the new release. I then have to start mixing versions of boost which is definitely a bad thing IMHO. Hence the reason I'm sticking with 1.30.2 and not moving to 1.31.0 And I'm not going to try and force or even request every boost library to support bcc32 (many already don't) such as lambda which I would love to use. The only way to stop the above happening is to force every current library to always support the compilers it supports now. That may severely hamper the development of the library which is a bad thing. Thanks Russell
Russell Hind <rhind@mac.com> writes:
And I'm not going to try and force or even request every boost library to support bcc32 (many already don't) such as lambda which I would love to use.
The only way to stop the above happening is to force every current library to always support the compilers it supports now. That may severely hamper the development of the library which is a bad thing.
Right now many Boost libraries are supporting lots of compiler (even older broken ones) with no force at all. I'm not convinced there's anything we can do here as a matter of policy. Individual library authors need to decide for themselves when it's time to drop support for various compilers. -- Dave Abrahams Boost Consulting www.boost-consulting.com
Bronek Kozicki <brok@rubikon.pl> writes:
On Tue, 17 Feb 2004 16:03:11 -0500, David Abrahams wrote:
Might be a good idea. I think I'd like to continue to support vc6 at least on the 1.31.x branch, particularly because last-minute changes in the graph library broke Boost.Python on vc6 for 1.31.0
If we agree to move in this direction, then in release 1.32 (or little later) we will be able to drop many workarounds from boost and simplify testing. But there is question that you raised:
What would we do about Borland, which is in some ways more broken than vc6? They don't ship a compiler I wouldn't consider broken.
I heard some voices that their new compiler being currently in beta (or maybe release candidate?) will be really close to standard. But we can be sure that old Borland compilers will still be widely in use, partialy due to lack of VCL support in this new compiler.
What about GCC 2.9Xes, which are standard equipment on some widely-used Linux distros? Way less broken than either of those two compilers, but still way out-of-date...
I would think that users of MSVC6, old Borland compilers or GCC 2.95 are "adoption blockers" and they possibly do not care much about new features in boost
Oh, but they do! Lots of these people just want to use Boost to get a job done. They don't give a flying hoo-hah what Boost itself needs to do to get *its* job done. For example, people from all over are picking up Boost.Python + Pyste and expecting to be able to wrap vc6 code with it. These people have every reason to want the same features I'd provide to everyone else if I could find a few more minutes to work on the library ;-) [The Python Windows distro is still built with vc6 (I don't expect that to change soon), and the Python developers will tell you that if you want to build reliable extension modules, you *have* to use vc6. Depending on what you intend to do with these extension modules, they're right] Another example: I'm sure vc6 users will care about unicode filename support in the filesystem library when/if we supply it.
(apparently they also do not care much about decent implementation of C++ standard library).
Probably not.
Thus hopefully they will not mind if we just ask them to use releases in 1.31.x branch. But it also means that we will need to support two branches for (at least) next two years. This might be cumbersome, but on the other hand it should simplify development in "current" branch.
Yep. It'll be cumbersome, that's for sure. Would it be any better than bringing in new "non-vc6" features with #ifdefs and new non-vc6 libraries without apology? -- Dave Abrahams Boost Consulting www.boost-consulting.com
"Bronek Kozicki" <brok@rubikon.pl> wrote in message news:1iw8nsmc4skor.1t4umw7rdiz3i.dlg@40tude.net...
I would think that users of MSVC6, old Borland compilers or GCC 2.95 are "adoption blockers" and they possibly do not care much about new features in boost (apparently they also do not care much about decent implementation of C++ standard library). Thus hopefully they will not mind if we just ask them to use releases in 1.31.x branch. But it also means that we will need to support two branches for (at least) next two years. This might be cumbersome, but on the other hand it should simplify development in "current" branch.
An important thing to realize is that programmers are not always free to choose their compilers. I agree in theory that dropping support for old problem compilers is a good idea; unfortunately, in practice, I am stuck using MSVC6 for a while longer. That doesn't mean I don't care about new features in Boost; quite the opposite: I push for a new compiler whenever possible so that I can use features *already* in Boost that don't work well with MSVC6. Mike
B.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Russell Hind <rhind@mac.com> writes:
David Abrahams wrote:
I expect that dropping vc6 support in Boost.Python in the next couple of years would significantly reduce its audience, so I can't justify it, (maybe not until Python's standard Windows distribution drops VC6 also). That fact has caused me to implement vc6 support in other libraries that Boost.Python depends on. Thereafter, if the library breaks, it's a regression. It's viral :(
Yes, but we have a very functional set of libraries in boost-1.30.2 and boost-1.31.0 so why not say older compilers are stuck to using these versions of boost? We're not saying they can't use boost, just not the latest version?
Might be a good idea. I think I'd like to continue to support vc6 at least on the 1.31.x branch, particularly because last-minute changes in the graph library broke Boost.Python on vc6 for 1.31.0 What would we do about Borland, which is in some ways more broken than vc6? They don't ship a compiler I wouldn't consider broken. What about GCC 2.9Xes, which are standard equipment on some widely-used Linux distros? Way less broken than either of those two compilers, but still way out-of-date... -- Dave Abrahams Boost Consulting www.boost-consulting.com
Beman Dawes <bdawes@acm.org> writes:
At 08:30 AM 2/12/2004, David Abrahams wrote:
Beman Dawes <bdawes@acm.org> writes:
Anyhow, I think your point about multiple reporting is a good one. The volume of tests is just too high. Fewer, more comprehensive, tests would be easier to monitor. Also fewer compilers. Do we really need to test every version of GCC and VC++ for the last four years?
Yes, IMO, if people want to support those compilers, we do need to test them.
It comes down to resources. I suppose if people really care about older compilers, they will be willing to contribute testing.
Yes. And as far as I can tell, they have been. meta-comm has been running the tests with vc6 and vc7, and Martin Wille (?) has been running the tests with nearly all versions of GCC ever produced ;^)
It would be easier to know if any particular tests was a resource problem if there was a bjam option to time the build and run aspects of the tests. The change Jeremy made recently to one or two of the graph tests noticeably sped testing, but it was only happenstance that I realized they were problems.
Yeah; it's doable but as usual it's a question of resources. When the amount of time we waste by having problems like this one becomes painful enough, though, I guess we'll all be willing to devote those resources ;-)
Human monitoring is just too error-prone. Why risk it? Why not have comprehensive tests with automated notifications when something breaks? It seems to me that less testing can only result in fewer things working, and coupled with human monitoring it's just going to make things worse, not better.
Great! Let's give it a try. Who has to do what to start?
Someone has to get BuildBot up and running. I'm too busy until at least 3/1. Rene said he might be able to try it earlier, but he only really has a sizeable chunk of time starting 3/1 as well. If someone else would like to do the community a great service, (s)he can be a pioneer: http://buildbot.sf.net -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
Beman Dawes <bdawes@acm.org> writes:
Great! Let's give it a try. Who has to do what to start?
Someone has to get BuildBot up and running. I'm too busy until at least 3/1. Rene said he might be able to try it earlier, but he only really has a sizeable chunk of time starting 3/1 as well.
And I have done some work toward this already... I have the different parts of BuildBot installed. I'm in the process of making a configuration file (and hence setting up CVS to send the right kind of emails). Unfortunately this is the hard part. Documentation on BuildBot is a bit lacking, a familiar theme :-), so it's a bit hard to decipher what I need to do. And since Brian's also only has time to help after 3/1 it makes it even slower :-( All I can say is.. _patience_ ;-) -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com - 102708583/icq
David Abrahams writes:
I'm really unconvinced that we need smaller tests on fewer compilers. Human monitoring is just too error-prone. Why risk it? Why not have comprehensive tests with automated notifications when something breaks? It seems to me that less testing can only result in fewer things working, and coupled with human monitoring it's just going to make things worse, not better.
Agree completely. -- Aleksey Gurtovoy MetaCommunications Engineering
Beman Dawes writes:
No, actually I do look at the meta-comm tests. In fact I review every test on every platform. It takes quite a while.
FWIW, we also produce an issues list which enumerates problematic (unexpectedly or newly failing and not marked up) tests only -- http://tinyurl.com/3yu64. -- Aleksey Gurtovoy MetaCommunications Engineering
On Feb 13, 2004, at 3:06 AM, Aleksey Gurtovoy wrote:
Beman Dawes writes:
No, actually I do look at the meta-comm tests. In fact I review every test on every platform. It takes quite a while.
FWIW, we also produce an issues list which enumerates problematic (unexpectedly or newly failing and not marked up) tests only -- http://tinyurl.com/3yu64.
Is this page publisized? (linked to from somewhere on www.boost.org that is highly visible?) If not, it should be. I didn't know it existed until now. Cheers, Jeremy _______________________________________________ Jeremy Siek <jsiek@osl.iu.edu> http://www.osl.iu.edu/~jsiek Ph.D. Student, Indiana University Bloomington C++ Booster (http://www.boost.org) Office phone: (812) 856-1820 _______________________________________________
At 08:43 AM 2/11/2004, David Abrahams wrote:
Douglas Paul Gregor <gregod@cs.rpi.edu> writes:
On Tue, 10 Feb 2004, David Abrahams wrote:
We also ought to be thinking about releasing 1.31.1 ASAP.
Then we need to pull together a (short!) list of the fixes we want in 1.31.1. Graph and Python fixes, as you've said. I wouldn't mind getting the /CLR workaround into Function.
I don't think there's a specific fix needed for Python, once the graph library is fixed.
What about the fails and missings on the Linux tests? Are those acceptable? --Beman
Beman Dawes wrote:
At 08:43 AM 2/11/2004, David Abrahams wrote:
Douglas Paul Gregor <gregod@cs.rpi.edu> writes:
On Tue, 10 Feb 2004, David Abrahams wrote:
We also ought to be thinking about releasing 1.31.1 ASAP.
There is at least one Boost.Build fix I did that should go in to that. -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com - 102708583/icq
Beman Dawes <bdawes@acm.org> writes:
At 08:43 AM 2/11/2004, David Abrahams wrote:
Douglas Paul Gregor <gregod@cs.rpi.edu> writes:
On Tue, 10 Feb 2004, David Abrahams wrote:
We also ought to be thinking about releasing 1.31.1 ASAP.
Then we need to pull together a (short!) list of the fixes we want in 1.31.1. Graph and Python fixes, as you've said. I wouldn't mind getting the /CLR workaround into Function.
I don't think there's a specific fix needed for Python, once the graph library is fixed.
What about the fails and missings on the Linux tests? Are those acceptable?
No, and they're very surprising to me. AFAIK those tests were all or mostly passing when I went away for business a week ago. I did watch my email and the Boost list, but I didn't see any alarms about these. Did I miss something? The regression logs don't seem to show the output for failed library builds, which seems to be the problem behind the "missing" tests. -- Dave Abrahams Boost Consulting www.boost-consulting.com
At 12:20 PM 2/10/2004, David Abrahams wrote:
IIUC, we've already found some regressions with 1.31.0, having to do with the graph library and MSVC6. These regressions affect the Python lib also, and the problems show up in the regression test results. It's very disappointing to me that even though the Python library was being tested continuously, we let the release go out anyway.
Yes, that worried me too. Wasn't Python a problem with the last major release too? Perhaps Python regressions should be looked at more often.
We ought to ask ourselves how this happened and see if there's something we can do to prevent it in the future.
It comes down to release discipline. If fixes aren't forthcoming for libraries that many other Boost libraries depend on, it delays testing on the other libraries. It was also pretty amazing how many commits were made not just after branch-for-release, but after release candidate testing started. It seems hard to motivate some developers until late in the process.
My guess is that inspecting regression results by hand isn't good enough, and we need an automated system like the BuildBot.
It's not like we didn't know about various regressions. And people doing the regression tests often posted messages indicating concern over failures. However, if we can turn around critical tests much more often, that would certainly help.
We also ought to be thinking about releasing 1.31.1 ASAP.
Go for it! It would be nice to resolve the issue of path lengths over 100 characters in tar files, too. --Beman
Beman Dawes <bdawes@acm.org> writes:
At 12:20 PM 2/10/2004, David Abrahams wrote:
IIUC, we've already found some regressions with 1.31.0, having to do with the graph library and MSVC6. These regressions affect the Python lib also, and the problems show up in the regression test results. It's very disappointing to me that even though the Python library was being tested continuously, we let the release go out anyway.
Yes, that worried me too. Wasn't Python a problem with the last major release too?
I don't know.
Perhaps Python regressions should be looked at more often.
Maybe. The Python library does depend on more parts of Boost than many others, so it makes a good "canary". Mistakes in other libraries can easily break it. But with Python tests not included in some of our "official" tests (ahem), we can easily miss a problem when it shows up.
We ought to ask ourselves how this happened and see if there's something we can do to prevent it in the future.
It comes down to release discipline.
Yes. When Meta-comm took over the 1.30.2 release they insisted on zero regressions on selected platforms/compilers. In the end it was a good thing.
If fixes aren't forthcoming for libraries that many other Boost libraries depend on, it delays testing on the other libraries.
Was that really a factor in this last release? Which libraries were delayed, and which fixes weren't forthcoming?
It was also pretty amazing how many commits were made not just after branch-for-release, but after release candidate testing started. It seems hard to motivate some developers until late in the process.
My guess is that inspecting regression results by hand isn't good enough, and we need an automated system like the BuildBot.
It's not like we didn't know about various regressions.
There was *no* information about the Python regression posted; at least, I didn't see any.
And people doing the regression tests often posted messages indicating concern over failures.
I thought they were usually addressed promptly.
However, if we can turn around critical tests much more often, that would certainly help.
We also ought to be thinking about releasing 1.31.1 ASAP.
Go for it!
Gadzooks; there's no possible way I can manage anything until at _least_ the beginning of March. Is there anyone else that can handle this?
It would be nice to resolve the issue of path lengths over 100 characters in tar files, too.
I guess so. I don't know much about the issue. -- Dave Abrahams Boost Consulting www.boost-consulting.com
At 11:16 AM 2/11/2004, David Abrahams wrote:
If fixes aren't forthcoming for libraries that many other Boost libraries depend on, it delays testing on the other libraries.
Was that really a factor in this last release? Which libraries were delayed, and which fixes weren't forthcoming?
Iterator adaptors and type traits hurt a lot when they break. Sometimes it takes awhile to figure out exactly what the problem is, too. For example, a change around January 16 caused VC++6.0 to ICE on several libraries. It didn't get fixed until the 26th, because it took time to realize where the fault lay. IIRC, it got fixed the same day once we had figured out what change actually triggered the failure. Earlier in the release cycle, it took a while to get the auto link stuff working on all compilers. The release manager can only focus on so many things at once. --Beman
Any plans on adding support for parsing wchar_t in the program_options lib? It is hard coded with std::string right now which makes it kind of useless in a program where argv is a wchar_t**. It could be templatized so that it can be based on any kind of std::basic_string. Alternatively, you could use preprocessor defines to control whether it builds for ANSI or UNICODE. This would probably be adequate for its intended use as well as less complicated to implement but I suspect it would not be portable enough for BOOST. Regards, Bryan Leppard Neotechnology Consultants Ltd.
Hi Bryan,
Any plans on adding support for parsing wchar_t in the program_options lib? It is hard coded with std::string right now which makes it kind of useless in a program where argv is a wchar_t**.
The review manager indicated that unicode (~= whar_t) support is needed for the final version of library, so yes, there are such plans. The problem is that it's probably the only planned change for which I haven't yet decided the design.
It could be templatized so that it can be based on any kind of std::basic_string.
Alas, that would break ability to put all sources in a library -- thereby increasing compile time for users who don't need wchar_t support.
Alternatively, you could use preprocessor defines to control whether it builds for ANSI or UNICODE. This would probably be adequate for its intended use as well as less complicated to implement but I suspect it would not be portable enough for BOOST.
That's an option too. Could you answer a couple a questions to help me with design: 1. Do you need wchar_t for option values, or for option names too? 2. Do you plan to mix char and wchar_t? If so, how 'char' is to be converted to wchar_t? What if I require to provide a converting function for that case? 3. If you plan to use wchar_t for option names, how will use specify names in code? Using unicode literals or using some specific 8-bit encoding? If the latter, how convertion to unicode is to be done? Again, is user-provided converter okay for you? Thanks, Volodya
Hi Volodya, On Sat, Feb 14, 2004 at 07:29:00PM +0300, Vladimir Prus wrote:
It could be templatized so that it can be based on any kind of std::basic_string.
Alas, that would break ability to put all sources in a library -- thereby increasing compile time for users who don't need wchar_t support.
AFAIR, there were several solutions for this problem. I don't see many other reasonable options besides char and wchar_t. So why don't just allow these specializations. Then you can put them into cpp files. std::string in STL is handled in very similar way. Regards, Pavol
Hi Pavol,
On Sat, Feb 14, 2004 at 07:29:00PM +0300, Vladimir Prus wrote:
It could be templatized so that it can be based on any kind of std::basic_string.
Alas, that would break ability to put all sources in a library -- thereby increasing compile time for users who don't need wchar_t support.
AFAIR, there were several solutions for this problem.
I don't see many other reasonable options besides char and wchar_t. So why don't just allow these specializations. Then you can put them into cpp files.
std::string in STL is handled in very similar way.
There's other very attactive option. If most of the time when user wants wchar_t he also need to specify 8bit encoding -> unicode convertion, it's possible to *always* use wchar_t and convert it to char only on interface boundaries. That's similiar to QT approach -- where there's single string class. - Volodya
On Sat, Feb 14, 2004 at 10:21:06PM +0300, Vladimir Prus wrote:
Hi Pavol,
On Sat, Feb 14, 2004 at 07:29:00PM +0300, Vladimir Prus wrote:
It could be templatized so that it can be based on any kind of std::basic_string.
Alas, that would break ability to put all sources in a library -- thereby increasing compile time for users who don't need wchar_t support.
AFAIR, there were several solutions for this problem.
I don't see many other reasonable options besides char and wchar_t. So why don't just allow these specializations. Then you can put them into cpp files.
std::string in STL is handled in very similar way.
There's other very attactive option. If most of the time when user wants wchar_t he also need to specify 8bit encoding -> unicode convertion, it's possible to *always* use wchar_t and convert it to char only on interface boundaries.
That's similiar to QT approach -- where there's single string class.
I can't judge rigthfuly, if this is a feasible approach. From my experience, I use to work with wchar_t on windows in gui applications, while linux based console program ususaly use only char (non-unicode) encoding. So it seems, that both char and wchar_t usage is equaly important. Your approach would penalize "char" approach in the means of additional conversions and memory requiremnts. Maybe it does not matter too much, but you should take this argument into consideration. Regards, Pavol
Any plans on adding support for parsing wchar_t in the program_options
I don't know how important conversions are to other people. When I write a Unicode program it is pure. I expect to use all Unicode characters, strings, and literals. I expect that I will have to handle any required conversions outside the library. I agree with the suggestion to set it up with just the specializations for char and wchar_t similar to std::string and std::wstring. Regards, Bryan Leppard Neotechnology Consultants Ltd. -----Original Message----- From: Vladimir Prus [mailto:ghost@cs.msu.su] Sent: February 14, 2004 9:29 AM To: Boost mailing list; Bryan Leppard Subject: Re: [boost] program_options wchar_t support Hi Bryan, lib?
It is hard coded with std::string right now which makes it kind of useless in a program where argv is a wchar_t**.
The review manager indicated that unicode (~= whar_t) support is needed for the final version of library, so yes, there are such plans. The problem is that it's probably the only planned change for which I haven't yet decided the design.
It could be templatized so that it can be based on any kind of std::basic_string.
Alas, that would break ability to put all sources in a library -- thereby increasing compile time for users who don't need wchar_t support.
Alternatively, you could use preprocessor defines to control whether it builds for ANSI or UNICODE. This would probably be adequate for its intended use as well as less complicated to implement but I suspect it would not be portable enough for BOOST.
That's an option too. Could you answer a couple a questions to help me with design: 1. Do you need wchar_t for option values, or for option names too? 2. Do you plan to mix char and wchar_t? If so, how 'char' is to be converted to wchar_t? What if I require to provide a converting function for that case? 3. If you plan to use wchar_t for option names, how will use specify names in code? Using unicode literals or using some specific 8-bit encoding? If the latter, how convertion to unicode is to be done? Again, is user-provided converter okay for you? Thanks, Volodya
Any plans on adding support for parsing wchar_t in the program_options
If you are interested, I have jury-rigged program_options so that you can switch between building it for Ansi or Unicode. Unfortunately it will probably only work for VC 7.1 but it demonstrates one approach. It serves my purpose for now but I don't think that this approach is appropriate for a Boost library. However, it may be a starting point for conversion to templates since I had to use aliases for all occurrences of char and std::string. Regards, Bryan Leppard -----Original Message----- From: Vladimir Prus [mailto:ghost@cs.msu.su] Sent: February 14, 2004 9:29 AM To: Boost mailing list; Bryan Leppard Subject: Re: [boost] program_options wchar_t support Hi Bryan, lib?
It is hard coded with std::string right now which makes it kind of useless in a program where argv is a wchar_t**.
The review manager indicated that unicode (~= whar_t) support is needed for the final version of library, so yes, there are such plans. The problem is that it's probably the only planned change for which I haven't yet decided the design.
It could be templatized so that it can be based on any kind of std::basic_string.
Alas, that would break ability to put all sources in a library -- thereby increasing compile time for users who don't need wchar_t support.
Alternatively, you could use preprocessor defines to control whether it builds for ANSI or UNICODE. This would probably be adequate for its intended use as well as less complicated to implement but I suspect it would not be portable enough for BOOST.
That's an option too. Could you answer a couple a questions to help me with design: 1. Do you need wchar_t for option values, or for option names too? 2. Do you plan to mix char and wchar_t? If so, how 'char' is to be converted to wchar_t? What if I require to provide a converting function for that case? 3. If you plan to use wchar_t for option names, how will use specify names in code? Using unicode literals or using some specific 8-bit encoding? If the latter, how convertion to unicode is to be done? Again, is user-provided converter okay for you? Thanks, Volodya
On Wed, 11 Feb 2004 17:20:36 -0500, Beman Dawes wrote
At 11:16 AM 2/11/2004, David Abrahams wrote:
If fixes aren't forthcoming for libraries that many other Boost libraries depend on, it delays testing on the other libraries.
Was that really a factor in this last release? Which libraries were >delayed, and which fixes weren't forthcoming?
Iterator adaptors and type traits hurt a lot when they break. Sometimes it takes awhile to figure out exactly what the problem is, too. For example, a change around January 16 caused VC++6.0 to ICE on several libraries. It didn't get fixed until the 26th, because it took time to realize where the fault lay. IIRC, it got fixed the same day once we had figured out what change actually triggered the failure. Earlier in the release cycle, it took a while to get the auto link stuff working on all compilers. The release manager can only focus on so many things at once.
To me the autolink thing was sort of an 'unplanned feature' for the release. I don't recall seeing any warning or discussion about it until we were just putting it in. Now, mind you John really did most of the work, which I appreciate greatly, but in my mind we should have left it out for this release because it is too destabalizing a change after the branch for release. In fact, right now is the time for destabalizing features to go in. I believe we have a couple new libraries to add, etc. In a couple months we ought to call a halt to all this sort of destabalizing change on the main branch. If people have exploritory changes they can move off to a development branch, or we can branch for release and stabalize it, and let the destabalizing change go on main. And that's the other key, once we branch for release feature development has to stop on the release candidate -- bug fixes only....I thought that was the policy, but it didn't happen that way... Jeff
To me the autolink thing was sort of an 'unplanned feature' for the release. I don't recall seeing any warning or discussion about it until we were just putting it in. Now, mind you John really did most of the work, which I appreciate greatly, but in my mind we should have left it out for this release because it is too destabalizing a change after the branch for release.
That's true, we/I got bounced into it by user requests (and maybe the usual excess of enthusiasm as well). BTW, next time we should set a clear timeline for what changes are acceptable when, with plenty of notice in advance etc... John.
Beman Dawes <bdawes@acm.org> writes:
At 11:16 AM 2/11/2004, David Abrahams wrote:
If fixes aren't forthcoming for libraries that many other Boost libraries depend on, it delays testing on the other libraries.
Was that really a factor in this last release? Which libraries were delayed, and which fixes weren't forthcoming?
Iterator adaptors and type traits hurt a lot when they break.
Sometimes it takes awhile to figure out exactly what the problem is, too. For example, a change around January 16 caused VC++6.0 to ICE on several libraries. It didn't get fixed until the 26th, because it took time to realize where the fault lay.
I don't believe that the vc6/iterator_facade interaction caused any significant delays in testing other libraries; they were working fine, then they stopped working, and then they were working again.
IIRC, it got fixed the same day once we had figured out what change actually triggered the failure. Earlier in the release cycle, it took a while to get the auto link stuff working on all compilers. The release manager can only focus on so many things at once.
Sure; this shouldn't be up to the release manager. It's simply too much work and too error-prone. An automated system can identify precisely which checkin is responsible for breakage. Why leave it up to the release manager to notice the problem, and go back to do a laborious binary search, then harrass other people to deal with the problem? -- Dave Abrahams Boost Consulting www.boost-consulting.com
participants (25)
-
Aleksey Gurtovoy -
Beman Dawes -
Boost -
Bronek Kozicki -
Bryan Leppard -
David Abrahams -
Douglas Gregor -
Douglas Paul Gregor
-
hartmutkaiser＠t-online.de -
Jeff Garland -
Jeremy Siek -
Jody Hagins -
John Maddock -
Martin Wille -
Michael Glassford -
Misha Bergal -
Pavel Vozenilek -
Pavol Droba -
Peter Dimov -
Raoul Gough -
Rene Rivera -
Russell Hind -
Samuel Krempp -
Victor A. Wagner, Jr. -
Vladimir Prus