Pull request sanitise run on Travis CI (Re: [git] Near future.. How do we deal with git-native libraries?)
On 2 December 2013 15:39, Niall Douglas <s_sourceforge@nedprod.com> wrote:
Later on I think it very likely a script might scan all pull requests to ensure their commits were made using the canonical Boost .gitattributes, and if they weren't to refuse the pull request.
GitHub pull requests (PR) are nicely handled by Travis CI service (https://travis-ci.org/). Perhaps, it would make sense to set up Travis for Boost repositories as well. Then, for every PR submitted to one of Boost repos, a Travis build is performed and PR sanitisation script could be part of that build. A nice feature of GitHub+Travis integration is that GitHub PR ticked will automatically display status of corresponding Travis build, warning about broken that particular PR is broken, so merge with caution, etc. Best regards, -- Mateusz Łoskot, http://mateusz.loskot.net
On 2 Dec 2013 at 16:20, Mateusz Loskot wrote:
Later on I think it very likely a script might scan all pull requests to ensure their commits were made using the canonical Boost .gitattributes, and if they weren't to refuse the pull request.
GitHub pull requests (PR) are nicely handled by Travis CI service (https://travis-ci.org/).
While a good idea, we can do a lot better again. I have a reasonable Jenkins + Travis + Coveralls implementation also working for every pull request to master branch. It took a while to get the kinks out, but you can see the pull request commentary at https://github.com/BoostGSoC/boost.afio/pull/48 where the bots decided if my "Boost v1.55 RC merge" pull request was valid.
Perhaps, it would make sense to set up Travis for Boost repositories as well. Then, for every PR submitted to one of Boost repos, a Travis build is performed and PR sanitisation script could be part of that build.
Given the work involved, and Travis's fairly restricted per job timeout, I think this will be a per-maintainer effort. It might be possible for a single "Does it build on Linux with default GCC?" sanity run yes, but for anything beyond that I fear it will be a per-project initiative. It took me many weeks to get AFIO's automated build infrastructure working right. I can't see anyone volunteering that time except for their own libraries.
A nice feature of GitHub+Travis integration is that GitHub PR ticked will automatically display status of corresponding Travis build, warning about broken that particular PR is broken, so merge with caution, etc.
While nice, it's too subtle for me. It also doesn't work if multiple bots are posting status to the same pull request, so if say the Windows unit test run succeeds but Linux fails, you have a 50/50 chance of seeing that the pull request has failed. This is a stupid limitation of github, where apparently the only possible CI target is one. While messy and generating a lot of unhelpful noise, I haven't found any way better than bot-posted comments so far. Unfortunate, but it does work. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
On 2 December 2013 17:48, Niall Douglas <s_sourceforge@nedprod.com> wrote:
On 2 Dec 2013 at 16:20, Mateusz Loskot wrote:
Later on I think it very likely a script might scan all pull requests to ensure their commits were made using the canonical Boost .gitattributes, and if they weren't to refuse the pull request.
GitHub pull requests (PR) are nicely handled by Travis CI service (https://travis-ci.org/).
While a good idea, we can do a lot better again.
I have a reasonable Jenkins + Travis + Coveralls implementation also working for every pull request to master branch. It took a while to get the kinks out, but you can see the pull request commentary at https://github.com/BoostGSoC/boost.afio/pull/48 where the bots decided if my "Boost v1.55 RC merge" pull request was valid.
Interesting, I'll take a look at that PR.
Perhaps, it would make sense to set up Travis for Boost repositories as well. Then, for every PR submitted to one of Boost repos, a Travis build is performed and PR sanitisation script could be part of that build.
Given the work involved, and Travis's fairly restricted per job timeout, I think this will be a per-maintainer effort. It might be possible for a single "Does it build on Linux with default GCC?" sanity run yes, but for anything beyond that I fear it will be a per-project initiative. It took me many weeks to get AFIO's automated build infrastructure working right. I can't see anyone volunteering that time except for their own libraries.
I should have explained my idea clearer, I didn't mean to use Travis CI for actual builds (compilation+ linking), because there are certain limitations, as you pointed. I meant to use Travis CI for some support lightweight tasks such as sanitising PRs, running Inspect, and perhaps hook-like things. Simply, to use Travis as Unix shell, not for running actual builds or regression tests. I may be stretching the purpose of Travis, I realise :)
A nice feature of GitHub+Travis integration is that GitHub PR ticked will automatically display status of corresponding Travis build, warning about broken that particular PR is broken, so merge with caution, etc.
While nice, it's too subtle for me. It also doesn't work if multiple bots are posting status to the same pull request, so if say the Windows unit test run succeeds but Linux fails, you have a 50/50 chance of seeing that the pull request has failed. This is a stupid limitation of github, where apparently the only possible CI target is one.
I see, I didn't realise that.
While messy and generating a lot of unhelpful noise, I haven't found any way better than bot-posted comments so far. Unfortunate, but it does work.
Makes sense to me. Best regards, -- Mateusz Łoskot, http://mateusz.loskot.net
On 2 Dec 2013 at 17:59, Mateusz Loskot wrote:
Given the work involved, and Travis's fairly restricted per job timeout, I think this will be a per-maintainer effort. It might be possible for a single "Does it build on Linux with default GCC?" sanity run yes, but for anything beyond that I fear it will be a per-project initiative. It took me many weeks to get AFIO's automated build infrastructure working right. I can't see anyone volunteering that time except for their own libraries.
I should have explained my idea clearer, I didn't mean to use Travis CI for actual builds (compilation+ linking), because there are certain limitations, as you pointed.
I meant to use Travis CI for some support lightweight tasks such as sanitising PRs, running Inspect, and perhaps hook-like things. Simply, to use Travis as Unix shell, not for running actual builds or regression tests.
Oh sure. Travis is very flexible. For example I have a job on there whose sole purpose is the run the unit tests with gcov and upload the coverage to Coveralls.io, because coveralls.io has special support for Travis. My only irritation with Travis is there is no such thing as job dependencies, so all jobs always run with each commit. On Jenkins I have a clang static analysis preflight check off which all other jobs hang as a dependency, so if the static analysis fails I know I've done something really stupid e.g. forgot to commit a file.
I may be stretching the purpose of Travis, I realise :)
If a commit could be rejected because Travis says no, I think that would a hugely useful feature. That isn't available to us, so sure a pull request automated scanner and rejector looks the next best thing. It just requires, I suppose, someone to do the work. I guess I have much more experience here than I should thanks to AFIO. Sometime soon after this modularisation we're surely going to have to add dependencies support to Boost modules so say Boost.AFIO can say it really needs Boost.Filesystem for example. Right now we have to rely on build failures to spot missing dependencies. It isn't ideal, especially for Travis as you have to hardcode the git clone of the right submodules as pulling the kitchen sink takes too long. That sounds brittle, and not especially maintainable.
While messy and generating a lot of unhelpful noise, I haven't found any way better than bot-posted comments so far. Unfortunate, but it does work.
Makes sense to me.
The problem with bot commenting is you, and everyone else in the project, gets an email for every comment. This because very wearing after a while, puts you off making pull requests :( Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
On 4 December 2013 17:52, Niall Douglas <s_sourceforge@nedprod.com> wrote:
On 2 Dec 2013 at 17:59, Mateusz Loskot wrote:
Given the work involved, and Travis's fairly restricted per job timeout, I think this will be a per-maintainer effort. It might be possible for a single "Does it build on Linux with default GCC?" sanity run yes, but for anything beyond that I fear it will be a per-project initiative. It took me many weeks to get AFIO's automated build infrastructure working right. I can't see anyone volunteering that time except for their own libraries.
I should have explained my idea clearer, I didn't mean to use Travis CI for actual builds (compilation+ linking), because there are certain limitations, as you pointed.
I meant to use Travis CI for some support lightweight tasks such as sanitising PRs, running Inspect, and perhaps hook-like things. Simply, to use Travis as Unix shell, not for running actual builds or regression tests.
Oh sure. Travis is very flexible. For example I have a job on there whose sole purpose is the run the unit tests with gcov and upload the coverage to Coveralls.io, because coveralls.io has special support for Travis. My only irritation with Travis is there is no such thing as job dependencies, so all jobs always run with each commit.
You're right, for such a complex project like Boost, that is quite a limitation.
I may be stretching the purpose of Travis, I realise :)
If a commit could be rejected because Travis says no, I think that would a hugely useful feature. That isn't available to us, so sure a pull request automated scanner and rejector looks the next best thing. It just requires, I suppose, someone to do the work. I guess I have much more experience here than I should thanks to AFIO.
I also have experience with Travis, configured it for a bunch of C++ projects, each with numerous and sometimes complex dependencies. So, if we see Travis useful for Boost at some point, I may help as well.
Sometime soon after this modularisation we're surely going to have to add dependencies support to Boost modules so say Boost.AFIO can say it really needs Boost.Filesystem for example. Right now we have to rely on build failures to spot missing dependencies. It isn't ideal, especially for Travis as you have to hardcode the git clone of the right submodules as pulling the kitchen sink takes too long. That sounds brittle, and not especially maintainable.
I think revisions matching is the common way to juggle submodules. That's what Qt does, I think:
From "Using latest branches in the submodules" at http://qt-project.org/wiki/Building_Qt_5_from_Git
"By default the checkout will not contain the latest stable/dev branches of each individual submodule repository, but a combination of versions that are known to work together."
While messy and generating a lot of unhelpful noise, I haven't found any way better than bot-posted comments so far. Unfortunate, but it does work.
Makes sense to me.
The problem with bot commenting is you, and everyone else in the project, gets an email for every comment. This because very wearing after a while, puts you off making pull requests :(
BTW, I at some point for SOCI project, was playing with sending GitHub notifications to a dedicated mailing list, subject lines prepared to indicate matter (i.e. comment or new issue, what label of issue) Such system allows use of e-mail inbox filtering and avoid unnecessary noise. I stole this idea from libusbx folks: http://pete.akeo.ie/2012/07/using-rss2email-and-github-feeds-to.html Best regards, -- Mateusz Łoskot, http://mateusz.loskot.net
Mateusz Loskot <mateusz@loskot.net> writes:
On 4 December 2013 17:52, Niall Douglas <s_sourceforge@nedprod.com> wrote:
On 2 Dec 2013 at 17:59, Mateusz Loskot wrote:
Given the work involved, and Travis's fairly restricted per job timeout, I think this will be a per-maintainer effort. It might be possible for a single "Does it build on Linux with default GCC?" sanity run yes, but for anything beyond that I fear it will be a per-project initiative. It took me many weeks to get AFIO's automated build infrastructure working right. I can't see anyone volunteering that time except for their own libraries.
I should have explained my idea clearer, I didn't mean to use Travis CI for actual builds (compilation+ linking), because there are certain limitations, as you pointed.
I meant to use Travis CI for some support lightweight tasks such as sanitising PRs, running Inspect, and perhaps hook-like things. Simply, to use Travis as Unix shell, not for running actual builds or regression tests.
Oh sure. Travis is very flexible. For example I have a job on there whose sole purpose is the run the unit tests with gcov and upload the coverage to Coveralls.io, because coveralls.io has special support for Travis. My only irritation with Travis is there is no such thing as job dependencies, so all jobs always run with each commit.
You're right, for such a complex project like Boost, that is quite a limitation.
Travis testing of boost projects should generally be done against the latest release of all the other projects, so it wouldn't really be that bad.
participants (3)
-
Dave Abrahams -
Mateusz Loskot -
Niall Douglas