[Boost.Test] New testing procedure?
According to http://tinyurl.com/3f32c tests have been failing (IIUC for several days if not longer) because of protected/private access violations in the test library. See the end of http://tinyurl.com/35ole for details. IMO this situation is unacceptable, and I'm looking for practical solutions. Problems with the Boost.Test affect our ability to get meaningful feedback on the state of other libraries. If Gennadiy can't test on more compilers before checking in, and respond faster to problems introduced in its source, and if library authors genuinely find Boost.Test useful, maybe we need to move to a different model wherein the test library's own tests are run on a CVS branch of the code (?) so that Gennadiy can see and deal with his problems before they are merged into the main trunk and break everything else? -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
David Abrahams writes:
According to http://tinyurl.com/3f32c tests have been failing (IIUC for several days if not longer) because of protected/private access violations in the test library. See the end of http://tinyurl.com/35ole for details. IMO this situation is unacceptable,
I agree. Note that the config subsystem is in the same position -- if it's broken, it breaks a lot of stuff all over the place. Not to say that it happens often, but still.
and I'm looking for practical solutions. Problems with the Boost.Test affect our ability to get meaningful feedback on the state of other libraries.
If Gennadiy can't test on more compilers before checking in, and respond faster to problems introduced in its source, and if library authors genuinely find Boost.Test useful, maybe we need to move to a different model wherein the test library's own tests are run on a CVS branch of the code (?) so that Gennadiy can see and deal with his problems before they are merged into the main trunk and break everything else?
The other regression runners have to speak for themselves, but we can definitely setup that here at Meta. As long as it's only the test and config libraries which will be tested on the branch, we have enough compilation resources to do that while preserving the twice-daily cycle for the main trunk (or the release branch). IMO we should go for it. -- Aleksey Gurtovoy MetaCommunications Engineering
Aleksey Gurtovoy wrote:
If Gennadiy can't test on more compilers before checking in, and respond faster to problems introduced in its source, and if library authors genuinely find Boost.Test useful, maybe we need to move to a different model wherein the test library's own tests are run on a CVS branch of the code (?) so that Gennadiy can see and deal with his problems before they are merged into the main trunk and break everything else?
The other regression runners have to speak for themselves, but we can definitely setup that here at Meta. As long as it's only the test and config libraries which will be tested on the branch, we have enough compilation resources to do that while preserving the twice-daily cycle for the main trunk (or the release branch).
IMO we should go for it.
OK for me too. I suppose this is not too difficult to automate too. Before going furhter let me tell you that I'm the guy running the regression tests on IBM, SGI and HP. There was some discussion on why thet were not update daily and some guessed that they were not automated. First of all: they are automated and normally run daily by means of a cron job. So why have'nt they run for quite some time now: well I fail to compile the regression test tools (exactly the reason why this thread started) on all platforms all of a sudden (well not all of a sudden as I don't verify the results every day). I'm currently in contact with IBM and HP to help me out and find workarounds or compiler updates to compile these tools succesfully. But this takes time. If you have questions about the regression results (and since I don't verify the results daily), you can always contact me (that's why I've put my name on the IBM regression test page for instance) and I will try to help you as good as I can. To answer another question in this thread: no, running the regression tests does not eat to much resources. They run every night as you also can see on the IBM regression test page. A big problem IMHO though is that few authors verify the regression status on all the different platforms. How could one otherwise explain that no one ever contacted me (even during this discussion) directly regarding some questions he might have regarding the regression tests (although my name is clearly visible on the status page for that sole purpose). This suggests that not many have ever taken a look at the status page for IBM/VisualAge. Actually only one developer ever contacted me asking for help to port his library succesfully to IBM/VisualAge. (don't get me wrong, I don't want to criticise boost developers who are all doing a great job but I'm just trying to indicate that it's not easy either to port the libraries). And again, I don't have the time every day to verify the regression tests and to read the ml (due to workload and/or traveling). So I usually find out if something breaks a week or two after the modification is made. And at that point, it's very difficult the exact reason why the test breaks almost having to reverse all changes until the situation where it worked and from then on trying to find a workaround. toon
participants (3)
-
Aleksey Gurtovoy -
David Abrahams -
Toon Knapen