Regression testing and BuildBot discussions
Dave Abrahams, Doug Gregor, Thomas Witt, and I had a discussion last week at the C++ committee meeting about how to improve Boost regression testing, and the whole process of getting ready for releases. Much of what we talked about has already been discussed on this list, so our conclusions shouldn't be much of a surprise: * Making Boost regression testing as robust, automatic, distributed, and timely as possible is the best way to help developers, release managers, testers, and eventually, users. * BuildBot (http://buildbot.sourceforge.net/) looks like best way to achieve those goals, based on experiments to date. * It would be very helpful if a team coalesces around Rene Rivera to develop BuildBot regression testing for Boost. They will need to stay coordinated with the MetaCommunications folks, of course. The rationale behind the "robust, automatic, and distributed" goals is that many people are willing to volunteer machine resources for Boost regression testing, but they say that other than downloading (and perhaps answering a very few startup wizard questions) they don't have any of their own time available. Thus the vision is of something like the SETI@home project which lets people contribute spare machine cycles without burdening them further. The rationale behind the "timely" goal is that developers will benefit tremendously from faster feedback, both for HEAD and active branches. Automating the feedback will free developers from having to track what is happening on many different platforms. The rationale behind encouraging a team to work on BuildBot regression testing is that this is important to Boost, so it will benefit from a team forming. That should improve and speed development, and ensure that several people understand the system and can maintain the internals. Rene is the logical leader since he has already stepped into that role for BuildBot. --Beman
The rationale behind the "timely" goal is that developers will benefit tremendously from faster feedback, both for HEAD and active branches. Automating the feedback will free developers from having to track what is happening on many different platforms.
Will it allow to test against non-HEAD branches? How would it work? Gennadiy
Gennadiy Rozental wrote:
The rationale behind the "timely" goal is that developers will benefit tremendously from faster feedback, both for HEAD and active branches. Automating the feedback will free developers from having to track what is happening on many different platforms.
Will it allow to test against non-HEAD branches? How would it work?
Perhaps the best response to the non-HEAD question is to post the configuration for the bot I'm currently running: config['builders'].append( { "name" : "RSI-SoL-Full", "slavename" : "RSI-SoL", "builddir" : "rsi_sol_full", "factory" : Boost_BuildFactory( action('cvs' ,root = ':ext:grafik@cvs.sourceforge.net:/cvsroot/boost' ,module = 'boost' ,files = [] ,haltOnFailure = True), action('bjam_build' ,haltOnFailure = True), action('btest_all' ,toolset = 'gcc-3_4_3-linux-i686-release' ,build = 'release' ,env = { 'PYTHON_ROOT': '/usr', 'PYTHON_VERSION': '2.3' } ,flunkOnFailure = False, warnOnFailure = True), action('test_tools_build' ,toolset = 'gcc' ,haltOnFailure = True), action('process_jam_log' ,haltOnFailure = True), action('collect_results' ,runner = 'RSI-SoL' ,haltOnFailure = True), action('publish_results' ,runner = 'RSI-SoL' ,publish_location = 'ftp://fx.meta-comm.com/boost-regression' ,haltOnFailure = True), workdir = 'boost', branch = 'HEAD', source_type = 'CVS' ) } ) As you can see testing on other branches is changing the "branch = HEAD" option above. Not that I've tried this yet :-) So it's likely to have initial problems. For those curious I added the code to the sandbox CVS under boost-sandbox/tools/boostbot/. -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim - Grafik/jabber.org
"Rene Rivera" <grafik.list@redshift-software.com> wrote in message news:42681042.2030709@redshift-software.com...
Gennadiy Rozental wrote:
The rationale behind the "timely" goal is that developers will benefit tremendously from faster feedback, both for HEAD and active branches. Automating the feedback will free developers from having to track what is happening on many different platforms.
Will it allow to test against non-HEAD branches? How would it work?
Perhaps the best response to the non-HEAD question is to post the configuration for the bot I'm currently running: [...] As you can see testing on other branches is changing the "branch = HEAD" option above. Not that I've tried this yet :-) So it's likely to have initial problems. For those curious I added the code to the sandbox CVS under boost-sandbox/tools/boostbot/.
Arhh, I hoped it would allow me to test my staff against development branch. Which doesn't seems to be the case since it's buildbot administrator who specify branch for whole tree, not a developer in hers Jamfile. Gennadiy
Gennadiy Rozental wrote:
Arhh, I hoped it would allow me to test my staff against development branch. Which doesn't seems to be the case since it's buildbot administrator who specify branch for whole tree, not a developer in hers Jamfile.
It's on my TODO to have that. I alluded to this on another post, but specifically I want to have OnDemand testers. They would allow a developer to go to the web page fill a form saying what test to run, what branch, and possibly a specific "version" to run. The code to have such a client is implemented, but the web page interface is not. Limited time and all that ;-) At minimum we can dedicate specific testers to test one branch. And people could use the current, launch build web page UI. -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim - Grafik/jabber.org
"Gennadiy Rozental" <gennadiy.rozental@thomson.com> writes:
"Rene Rivera" <grafik.list@redshift-software.com> wrote in message news:42681042.2030709@redshift-software.com...
Gennadiy Rozental wrote:
The rationale behind the "timely" goal is that developers will benefit tremendously from faster feedback, both for HEAD and active branches. Automating the feedback will free developers from having to track what is happening on many different platforms.
Will it allow to test against non-HEAD branches? How would it work?
Perhaps the best response to the non-HEAD question is to post the configuration for the bot I'm currently running: [...] As you can see testing on other branches is changing the "branch = HEAD" option above. Not that I've tried this yet :-) So it's likely to have initial problems. For those curious I added the code to the sandbox CVS under boost-sandbox/tools/boostbot/.
Arhh, I hoped it would allow me to test my staff against development branch. Which doesn't seems to be the case since it's buildbot administrator who specify branch for whole tree, not a developer in hers Jamfile.
I think developers need individual control over branch configuration testing, but the Jamfile is the wrong place to put it. For one thing, "test the HEAD (or the release branch if we have one)" has to be the default during a release. Branch testing for individual developers has to go on in parallel with the tests that check the overall health of the CVS HEAD. -- Dave Abrahams Boost Consulting www.boost-consulting.com
Beman Dawes wrote:
The rationale behind encouraging a team to work on BuildBot regression testing is that this is important to Boost, so it will benefit from a team forming. That should improve and speed development, and ensure that several people understand the system and can maintain the internals. Rene is the logical leader since he has already stepped into that role for BuildBot.
I see I've been volunteered :-) I guess if no one else ardently wants the job I'll do it. -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim - Grafik/jabber.org
participants (4)
-
Beman Dawes -
David Abrahams -
Gennadiy Rozental -
Rene Rivera