This is the summary of regression results as I see it: 0. The regression framework seems OK. It's also more reliable now -- thanks to Aleksey, who made XSLT processing more strict, and fixed process_jam_log not to loose some output. 1. We have a number of regressions with intel-linux-9.0. They come from two runners -- Martin and Sandia. In Martin's result most failures are due to an command line option that intel-9.0 does not understand, and Martin's results were not updated since I've checked in the fix. Sandia's results are post fix, and I see no linker errors. There are some failures that look genuine. 2. We have a large number of regression with msvc-stlport configurations. Most are linker errors, which suggests misconfiguration or Boost.Build bug, or both. I'll work with Roland on a solution. 3. There are some fails that look genuine. I think the right way to get 1.34 released now, as far as regression tests are concerned is this: 1. Completely freeze the list of tested compilers. This means that anything not tested now is not tested. Notably, we'll have no coverage for mingw or cygwin versions of gcc. 2. Have me and Roland fix stlport issues. 3. Start pinging developers about remaining failures. I think we should adopt a policy that will guarantee that all failures be resolved in a reasonable time -- namely, if a failure is not fixed by a certain date, it's marked expected and we move on. That cut-off date might be two weeks from the next Monday -- Feb 26. Both (1) and (3) will certainly have an effect on release quality -- we'll mark some failures as expected instead of fixing them, and we won't see failures on some configuration. However, I believe having a release sooner is more important at this point. Thomas, I would like you to express your opinion on this plan. Opinions from other Boosters are also appreciated! - Volodya
Vladimir Prus <ghost@cs.msu.su> writes:
1. Completely freeze the list of tested compilers. This means that anything not tested now is not tested. Notably, we'll have no coverage for mingw or cygwin versions of gcc.
I've been testing mingw. I was going to do a run over the weekend. Shall I now not bother? Anthony -- Anthony Williams Just Software Solutions Ltd - http://www.justsoftwaresolutions.co.uk Registered in England, Company Number 5478976. Registered Office: 15 Carrallack Mews, St Just, Cornwall, TR19 7UL
Anthony Williams wrote:
Vladimir Prus <ghost@cs.msu.su> writes:
1. Completely freeze the list of tested compilers. This means that anything not tested now is not tested. Notably, we'll have no coverage for mingw or cygwin versions of gcc.
I've been testing mingw. I was going to do a run over the weekend. Shall I now not bother?
I don't know. It's up to Thomas. An alternative solution might be this: - We start two week timer on fails we have now - We have mingw run - At the end of two weeks, unfixed fails we have now *and* any unfixed fails in mingw output gets marked as expected. - Volodya
Anthony Williams wrote:
I've been testing mingw. I was going to do a run over the weekend. Shall I now not bother?
I also already submitted gcc-3.4.2_mingw and gcc-3.4.5_mingw to th ftp server. The files will show up in the next table rebuild. I also think at least one cygwin toolset should be covered. Roland
Roland Schwarz wrote:
Anthony Williams wrote:
I've been testing mingw. I was going to do a run over the weekend. Shall I now not bother?
I also already submitted gcc-3.4.2_mingw and gcc-3.4.5_mingw to th ftp server. The files will show up in the next table rebuild.
I also think at least one cygwin toolset should be covered.
Thing is, I think we are, no offense intended, past the "should" point. As I said in my post to Anthony I am not going to stifle work on this but I am asking everybody to focus on what is there now and I am not going to hold the release for this. Thomas PS: If that's not clear I will be thankful if somebody makes it work -- Thomas Witt witt@acm.org
Anthony Anthony Williams wrote:
Vladimir Prus <ghost@cs.msu.su> writes:
1. Completely freeze the list of tested compilers. This means that anything not tested now is not tested. Notably, we'll have no coverage for mingw or cygwin versions of gcc.
I've been testing mingw. I was going to do a run over the weekend. Shall I now not bother?
It's more like we don't bother ;-). Seriously removing a compiler from the list of required platforms means is that we won't halt the release for regressions. We are nevertheless interested in results and I will accept fixes in the same way I will accept them for release platforms. HTH Thomas -- Thomas Witt witt@acm.org
Vladimir Prus ha escrito:
This is the summary of regression results as I see it:
0. The regression framework seems OK. It's also more reliable now -- thanks to Aleksey, who made XSLT processing more strict, and fixed process_jam_log not to loose some output.
1. We have a number of regressions with intel-linux-9.0. They come from two runners -- Martin and Sandia. In Martin's result most failures are due to an command line option that intel-9.0 does not understand, and Martin's results were not updated since I've checked in the fix. Sandia's results are post fix, and I see no linker errors. There are some failures that look genuine.
2. We have a large number of regression with msvc-stlport configurations. Most are linker errors, which suggests misconfiguration or Boost.Build bug, or both. I'll work with Roland on a solution.
3. There are some fails that look genuine.
I think the right way to get 1.34 released now, as far as regression tests are concerned is this:
1. Completely freeze the list of tested compilers. This means that anything not tested now is not tested. Notably, we'll have no coverage for mingw or cygwin versions of gcc.
2. Have me and Roland fix stlport issues.
3. Start pinging developers about remaining failures. I think we should adopt a policy that will guarantee that all failures be resolved in a reasonable time -- namely, if a failure is not fixed by a certain date, it's marked expected and we move on.
That cut-off date might be two weeks from the next Monday -- Feb 26.
Both (1) and (3) will certainly have an effect on release quality -- we'll mark some failures as expected instead of fixing them, and we won't see failures on some configuration. However, I believe having a release sooner is more important at this point.
Thomas, I would like you to express your opinion on this plan. Opinions from other Boosters are also appreciated!
I support this aggresive deadline-based approach and very much appreciate your efforts to get Boost 1.34 out of the door, we've just been procrastinating way too long. Hopefully, the planned post-1.34 switch to SVN and Beman's always-ready approach to code base management will avoid these problems in the future. Thank you, Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
Joaquín Mª López Muñoz wrote:
I support this aggresive deadline-based approach and very much appreciate your efforts to get Boost 1.34 out of the door, we've just been procrastinating way too long. Hopefully, the planned post-1.34 switch to SVN and Beman's always-ready approach to code base management will avoid these problems in the future.
In principle I agree. Where I do not agree is the fact that we currently do not have a single gcc (mingw/cygwin) regression result. I think it would be a fault if boost is omitting such a mainstream compiler! Roland
Roland Schwarz wrote:
Joaquín Mª López Muñoz wrote:
I support this aggresive deadline-based approach and very much appreciate your efforts to get Boost 1.34 out of the door, we've just been procrastinating way too long. Hopefully, the planned post-1.34 switch to SVN and Beman's always-ready approach to code base management will avoid these problems in the future.
In principle I agree. Where I do not agree is the fact that we currently do not have a single gcc (mingw/cygwin) regression result.
I think it would be a fault if boost is omitting such a mainstream compiler!
Nod. However, there are problems building dll's with mingw that are causing us apparent regressions (the static library builds all work - which is what we tested at the last release). See http://tinyurl.com/26pz5g for a typical example. The cause is a mingw linker problem rather than anything we can fix in Boost. I had a go at changing the regex/build Jamfile to static link when building with mingw, but apparently the toolset names don't match up: is there any way we can apply a build option to all gcc-mingw* toolsets? John.
Vladimir, First and foremost thanks to you and everybody else involved for all the work that went into this! Vladimir Prus wrote:
This is the summary of regression results as I see it:
I think the right way to get 1.34 released now, as far as regression tests are concerned is this:
1. Completely freeze the list of tested compilers. This means that anything not tested now is not tested. Notably, we'll have no coverage for mingw or cygwin versions of gcc.
Agreed. Are you going to update the list of toolsets in explicit-failures-markup.xml or should I do it?
2. Have me and Roland fix stlport issues.
Can you give us a heads-up when the config related issues are sorted out so that we have a chance to look at the remaining issues?
3. Start pinging developers about remaining failures. I think we should adopt a policy that will guarantee that all failures be resolved in a reasonable time -- namely, if a failure is not fixed by a certain date, it's marked expected and we move on.
Agreed.
That cut-off date might be two weeks from the next Monday -- Feb 26.
Agreed.
Both (1) and (3) will certainly have an effect on release quality -- we'll mark some failures as expected instead of fixing them, and we won't see failures on some configuration. However, I believe having a release sooner is more important at this point.
All too true. That being said I believe that the quality implications are probably negligible. Thomas -- Thomas Witt witt@acm.org
Thomas Witt wrote:
Vladimir Prus wrote:
This is the summary of regression results as I see it:
I think the right way to get 1.34 released now, as far as regression tests are concerned is this:
1. Completely freeze the list of tested compilers. This means that anything not tested now is not tested. Notably, we'll have no coverage for mingw or cygwin versions of gcc.
Agreed. Are you going to update the list of toolsets in explicit-failures-markup.xml or should I do it?
Please do.
2. Have me and Roland fix stlport issues.
Can you give us a heads-up when the config related issues are sorted out so that we have a chance to look at the remaining issues?
Roland is doing another test run now, with stlport build with native wchar_t. I suppose the results will show up soon.
3. Start pinging developers about remaining failures. I think we should adopt a policy that will guarantee that all failures be resolved in a reasonable time -- namely, if a failure is not fixed by a certain date, it's marked expected and we move on.
Agreed.
That cut-off date might be two weeks from the next Monday -- Feb 26.
Agreed.
Ok, so be it. For avoidance of doubt, but cut-off date I meant that any fails present at this date are marked as expected, and we completely freeze the code. In other words, this is date by which failures must be fixed to enter 1.34. Do you think a separate announcement to this effect is necessary, so that everybody notice it?
Both (1) and (3) will certainly have an effect on release quality -- we'll mark some failures as expected instead of fixing them, and we won't see failures on some configuration. However, I believe having a release sooner is more important at this point.
All too true. That being said I believe that the quality implications are probably negligible.
Perhaps. - Volodya
Thomas Witt wrote:
1. Completely freeze the list of tested compilers. This means that anything not tested now is not tested. Notably, we'll have no coverage for mingw or cygwin versions of gcc.
Agreed. Are you going to update the list of toolsets in explicit-failures-markup.xml or should I do it?
I have a concern here that Borland BCB6 was being tested by Metacomm, but their test results have not been uploaded since the 'freeze results' call a few weeks ago. Likewise, no-one else has been able to pick up the strain for the same reason. This was supposed to be the last supported release for BCB6, just as with MSVC6, and I would hate to lose it at the last minute. I have been patiently waiting for the regression runs to sort themselves out before chasing up a set of BCB6 results - I will re-install a copy and run the test myself if necessary - but it seems I can no longer afford to wait. For reference, BCB6 and BCB2006 were very similar in fails at the last count, with BCB6 being the main soure of expect-fail markups. Once the current Spirit/serialization issues are sorted out I do not anticipate further problems. -- AlisdairM
AlisdairM wrote:
Thomas Witt wrote:
1. Completely freeze the list of tested compilers. This means that anything not tested now is not tested. Notably, we'll have no coverage for mingw or cygwin versions of gcc.
Agreed. Are you going to update the list of toolsets in explicit-failures-markup.xml or should I do it?
I have a concern here that Borland BCB6 was being tested by Metacomm, but their test results have not been uploaded since the 'freeze results' call a few weeks ago. Likewise, no-one else has been able to pick up the strain for the same reason.
This was supposed to be the last supported release for BCB6, just as with MSVC6, and I would hate to lose it at the last minute.
What compiler version is BCB6? Is that 5.6.4 or something?
I have been patiently waiting for the regression runs to sort themselves out before chasing up a set of BCB6 results - I will re-install a copy and run the test myself if necessary - but it seems I can no longer afford to wait.
For reference, BCB6 and BCB2006 were very similar in fails at the last count, with BCB6 being the main soure of expect-fail markups. Once the current Spirit/serialization issues are sorted out I do not anticipate further problems.
They are sorted -- there's a procedure one can use to test borland with old spirit. - Volodya
AlisdairM wrote:
Yes, BCB6 == 5.6.4
I have an older one (5.5.1) still installed. (The free version). If you know where I can get a free download of the 5.6.4 I'll happily include this to my regressions. Else it would be nice if you could provide at least a run of yours. Roland
On Feb 9, 2007, at 2:24 AM, Vladimir Prus wrote:
This is the summary of regression results as I see it:
1. We have a number of regressions with intel-linux-9.0.
In Martin's result most failures are due to an command line option that intel-9.0 does not understand, and Martin's results were not updated since I've checked in the fix.
Martin, your latest run (Sat, 10 Feb 2007 22:42:11 +0000) still seems to have this same problem. -- Noel
K. Noel Belcourt wrote:
On Feb 9, 2007, at 2:24 AM, Vladimir Prus wrote:
This is the summary of regression results as I see it:
1. We have a number of regressions with intel-linux-9.0.
In Martin's result most failures are due to an command line option that intel-9.0 does not understand, and Martin's results were not updated since I've checked in the fix.
Martin, your latest run (Sat, 10 Feb 2007 22:42:11 +0000) still seems to have this same problem.
And linker errors still mention -h option that's supposed to be replaced with -soname in intel-linux.jam, so this suggests something was not updated. - Volodya
3. Start pinging developers about remaining failures. I think we should adopt a policy that will guarantee that all failures be resolved in a reasonable time -- namely, if a failure is not fixed by a certain date, it's marked expected and we move on.
Will this include the release failures, or just the debug testing? FWIW, I'd love to see a 1.34.0 with some of the warning issues fixed: there is a lot of noise in the boost 1.34.0 RC headers, especially with linux-based g++ compilers. Usually 1/2 of the vendor patch sets for boost are just warning cleanups. It would be great to get this fixed systematically upstream. best, benjamin
Vladimir Prus <ghost@cs.msu.su> writes:
1. We have a number of regressions with intel-linux-9.0. They come from two runners -- Martin and Sandia.
There also appears to be some kind of issue with Sandia's Python installation, or their gettext library. I can't tell exactly what the issue is, but these link errors are not Python library bugs. There may be something we can do in Boost.Build, but I need Sandia to tell us what (e.g. "link with libxxxx"). http://engineering.meta-comm.com/boost-regression/CVS-RC_1_34_0/developer/ou... -- Dave Abrahams Boost Consulting www.boost-consulting.com
On Mar 1, 2007, at 9:31 AM, David Abrahams wrote:
Vladimir Prus <ghost@cs.msu.su> writes:
1. We have a number of regressions with intel-linux-9.0. They come from two runners -- Martin and Sandia.
There also appears to be some kind of issue with Sandia's Python installation, or their gettext library. I can't tell exactly what the issue is, but these link errors are not Python library bugs. There may be something we can do in Boost.Build, but I need Sandia to tell us what (e.g. "link with libxxxx").
http://engineering.meta-comm.com/boost-regression/CVS-RC_1_34_0/ developer/output/Sandia-boost-bin-v2-libs-python-test-bases-test- intel-linux-9-0-debug_release.html
Hi Dave, It appears that all the missing symbols are in /usr/lib/ libreadline.so.4.3 on our system. -- Noel
Hi, The Sandia Intel 9.0 iostreams failures: http://preview.tinyurl.com/2gwtxp http://preview.tinyurl.com/2zljdv http://preview.tinyurl.com/327mdp are all configuration problems on our local system (we're missing the link from /usr/lib/libbz2.so to /usr/lib/libbz2.so.1.0.2). lrwxrwxrwx 1 root root 15 Nov 9 08:41 libbz2.so.1 -> libbz2.so. 1.0.2 -rwxr-xr-x 1 root root 70388 Dec 16 2005 libbz2.so.1.0.2 I've asked our system administrators to fix this problem but while I wait for them to get it fixed, I wanted to let you know that these problems are expected from the Sandia results and do not indicate problems with the boost tests. -- Noel
K. Noel Belcourt wrote:
Hi,
The Sandia Intel 9.0 iostreams failures:
http://preview.tinyurl.com/2gwtxp http://preview.tinyurl.com/2zljdv http://preview.tinyurl.com/327mdp
are all configuration problems on our local system (we're missing the link from /usr/lib/libbz2.so to /usr/lib/libbz2.so.1.0.2).
lrwxrwxrwx 1 root root 15 Nov 9 08:41 libbz2.so.1 -> libbz2.so. 1.0.2 -rwxr-xr-x 1 root root 70388 Dec 16 2005 libbz2.so.1.0.2
I've asked our system administrators to fix this problem but while I wait for them to get it fixed, I wanted to let you know that these problems are expected from the Sandia results and do not indicate problems with the boost tests.
Noel, unfortunately, you test results have the same toolset as Martin's results. Therefore, we cannot mark your failures, specifically, as expected. To avoid adding noise in regression results, perhaps you can stop running tests for RC_1_34_0, and instead start (now or after 1.34 release), running tests for HEAD, where we'll be more at position to address your configuration issues and any compiler-patch-level related failures? If so, I'll remove last set of your results. What do you think? - Volodya
Please remove my results and the results for Noel Belcourt that I accidently ran several weeks ago. I'll change the tests over to run against the HEAD. -- Noel On Mar 6, 2007, at 1:08 PM, Vladimir Prus wrote:
K. Noel Belcourt wrote:
Hi,
The Sandia Intel 9.0 iostreams failures:
http://preview.tinyurl.com/2gwtxp http://preview.tinyurl.com/2zljdv http://preview.tinyurl.com/327mdp
are all configuration problems on our local system (we're missing the link from /usr/lib/libbz2.so to /usr/lib/libbz2.so.1.0.2).
lrwxrwxrwx 1 root root 15 Nov 9 08:41 libbz2.so.1 -> libbz2.so. 1.0.2 -rwxr-xr-x 1 root root 70388 Dec 16 2005 libbz2.so.1.0.2
I've asked our system administrators to fix this problem but while I wait for them to get it fixed, I wanted to let you know that these problems are expected from the Sandia results and do not indicate problems with the boost tests.
Noel, unfortunately, you test results have the same toolset as Martin's results. Therefore, we cannot mark your failures, specifically, as expected.
To avoid adding noise in regression results, perhaps you can stop running tests for RC_1_34_0, and instead start (now or after 1.34 release), running tests for HEAD, where we'll be more at position to address your configuration issues and any compiler-patch-level related failures? If so, I'll remove last set of your results.
What do you think?
Please remove my results and the results for Noel Belcourt that I accidently ran several weeks ago. I'll change the tests over to run against the HEAD starting tonight. Thanks Volodya. -- Noel
participants (10)
-
AlisdairM -
Anthony Williams -
Benjamin Kosnik -
David Abrahams -
Joaquín Mª López Muñoz -
John Maddock -
K. Noel Belcourt -
Roland Schwarz -
Thomas Witt -
Vladimir Prus