TravisCI and Coverall usage policies in Boost
Hi, Motivation. I'd like to show some useful stuff using the Boost.Variant library as an example. Take a look at this link: https://coveralls.io/builds/1224886 It shows tests coverage and has a nice interface to get info about untested things. For example: * if we take a look at this https://coveralls.io/files/293662719 , we'll see that `polymorphic_get` was not tested for extracting type that is not in a variant variable * if we take a look at this https://coveralls.io/files/293662720 we'll see that swapping of recursive wrappers was not tested All that stuff was achieved by adding the https://github.com/apolukhin/variant/blob/travisci/.travis.yml file and turning a switch at coveralls.io site. Now let's take a look at the regression tests of variant: http://www.boost.org/development/tests/master/developer/variant.html . There we can see a indirect_travisci_tests tester. There's nothing special about that tester, except the fact that results are gathered using foreign TravisCI service. We can integrate foreign service to do regression reporting for us. We can even make this separately for each of the Boost libraries, triggering test run on every commit and reporting the results directly to README.md file (for example https://github.com/apolukhin/Boost.DLL#test-results). The problem. A few things make me nervous. First of all, TravisCI requires some access permissions to the repo. I'm not a github expert so I'm not 100% sure that this is safe. Second one, is that there is no policy in Boost about usage of foreign services like TravisCI and Coveralls. The questions. Is is OK to commit to Boost repos .travis.yml files without enabling TravisCI runs? Do we need a license note in .travis.yml file? Can we enable automated testing using TravisCI for a Boost repo? Can we enable automated tests coverage using Coverall for a Boost repo? -- Best regards, Antony Polukhin
On 18 Sep 2014 at 12:38, Antony Polukhin wrote:
A few things make me nervous.
First of all, TravisCI requires some access permissions to the repo. I'm not a github expert so I'm not 100% sure that this is safe.
Meh. If it were subversion, fair enough, but you can't delete stuff from git without it breaking all pushes and pulls to git. In other words, we'd notice, and every clone is also a backup. Travis could push botnet code into unit tests so CIs around the world execute it I suppose. I have my Jenkins CI configured with a level of paranoia verging on the extreme (no visibility of local machines or network, network is completely held separate - all Windows CI processes run at User perms, not Administrator), but that still allows a bot net.
Second one, is that there is no policy in Boost about usage of foreign services like TravisCI and Coveralls.
The questions.
Is is OK to commit to Boost repos .travis.yml files without enabling TravisCI runs?
Yes.
Do we need a license note in .travis.yml file?
Yes.
Can we enable automated testing using TravisCI for a Boost repo?
Yes. In fact it should be mandatory for all Boost libraries.
Can we enable automated tests coverage using Coverall for a Boost repo?
Yes. This should also be mandatory for all Boost libraries. It's free, nobody has an excuse. We should really add a Boost wiki page on setting this stuff up, and strongly hint at it being nearly a requirement on the community review page for new libraries. I can help write this if you'd like to start it Antony? Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
2014-09-18 12:53 GMT+04:00 Niall Douglas <s_sourceforge@nedprod.com>:
On 18 Sep 2014 at 12:38, Antony Polukhin wrote:
Can we enable automated tests coverage using Coverall for a Boost repo?
Yes. This should also be mandatory for all Boost libraries. It's free, nobody has an excuse.
We should really add a Boost wiki page on setting this stuff up, and strongly hint at it being nearly a requirement on the community review page for new libraries. I can help write this if you'd like to start it Antony?
Having such a wiki page would be good. I've finished writing a generic .travis.yml draft file that is suitable for almost any library that uses Boost: https://github.com/apolukhin/variant/blob/travisci/.travis.yml There's some 'sed' black magic to provide only current library files to coveralls site. Fast recursive git cloning is taken from run.py script. I've also made a draft of a README.md file with results table https://github.com/apolukhin/variant/tree/travisci. We could start writing the wiki. Do you know where to start? It would be good to hear more opinions about the TravisCI+Coveralls before we start to add .travis.yml files to all the libraries. -- Best regards, Antony Polukhin
On 18 Sep 2014 at 17:40, Antony Polukhin wrote:
Can we enable automated tests coverage using Coverall for a Boost repo?
Yes. This should also be mandatory for all Boost libraries. It's free, nobody has an excuse.
We should really add a Boost wiki page on setting this stuff up, and strongly hint at it being nearly a requirement on the community review page for new libraries. I can help write this if you'd like to start it Antony?
Having such a wiki page would be good.
I've finished writing a generic .travis.yml draft file that is suitable for almost any library that uses Boost: https://github.com/apolukhin/variant/blob/travisci/.travis.yml There's some 'sed' black magic to provide only current library files to coveralls site. Fast recursive git cloning is taken from run.py script.
You may find my Travis script inspirational at https://github.com/BoostGSoC13/boost.afio/blob/master/.travis.yml In particular, I find the time spent cloning all of Boost takes away valuable unit testing time especially when you're running valgrind or the thread sanitiser, so I keep an automatically updated copy of Boost releases at https://github.com/ned14/boost-release. What I do then on my Jenkins CI is to extract that only, delete from libs/ the libraries I want trunk for, and symbolic link in the trunk submodules just for those libraries I want trunk for. A quick b2 headers later and it's good. I also skip the Travis GEM and use curl :) Note my script is heavily based on Daniel Pfeifer's, so it's not all mine.
I've also made a draft of a README.md file with results table https://github.com/apolukhin/variant/tree/travisci.
Nice. Only other thing is you should really disambiguate by compiler version and platform. For example, https://boostgsoc13.github.io/boost.afio/.
We could start writing the wiki. Do you know where to start?
I guess go to https://svn.boost.org/trac/boost/wiki and start a page.
It would be good to hear more opinions about the TravisCI+Coveralls before we start to add .travis.yml files to all the libraries.
Yes, I think that where the full unit test library is being used that a summary of failing versus passing tests should be recorded. Coverage can be great, but if unit tests don't return failure for a problem it can get overlooked. For example, I patched Boost.Expected to spit out unit test results, and Jenkins makes this nice table: https://ci.nedprod.com/view/Boost%20Thread-Expected-Permit/job/Boost.E xpected%20Test%20Linux%20GCC%204.8/lastCompletedBuild/testReport/ I also think a valgrind pass needs to happen, plus a thread sanitiser pass, plus a clang static analysis pass. For all libraries. Unless they have big red fail marks all over them, libraries won't get fixed. If we ever get Windows on Travis, I was surprised how good the MSVC static analyser has become, and we should have a pass with that too. Niall Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
2014-09-19 2:46 GMT+04:00 Niall Douglas <s_sourceforge@nedprod.com>: <...>
You may find my Travis script inspirational at https://github.com/BoostGSoC13/boost.afio/blob/master/.travis.yml
<...>
Note my script is heavily based on Daniel Pfeifer's, so it's not all mine.
Yep, I've started my own script looking at yours. There's a problem with permissions. Current library maintainer permissions are insufficient to allow Travis builds for a boostorg library. In other words: I can not flick the Travis switch for any of the Boost libraries. This requires attention of one of the Boost admins. If we wish to allow Travis builds, then the admin must enable Travis builds for a particular library. A more generic solution would be to enable Travis builds for all the Boost libraries and set "Build only if .travis.yml is present" option to "on" for each project. Admins what do you think? -- Best regards, Antony Polukhin
On 24 Sep 2014 at 15:22, Antony Polukhin wrote:
There's a problem with permissions. Current library maintainer permissions are insufficient to allow Travis builds for a boostorg library. In other words: I can not flick the Travis switch for any of the Boost libraries.
This requires attention of one of the Boost admins. If we wish to allow Travis builds, then the admin must enable Travis builds for a particular library. A more generic solution would be to enable Travis builds for all the Boost libraries and set "Build only if .travis.yml is present" option to "on" for each project.
Admins what do you think?
I'd raise a pull request against boostorg for this, and post a message to the steering committee forum to ask for action with a link to the pull request. I'd say getting this approved should be a doozy. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
[Please do not mail me a copy of your followup] Antony Polukhin <antoshkka@gmail.com> spake the secret code <CAKqmYPb1p_Sz_mesSHFuqyJoRivk3KUX-Pikn2grnvVbzpJreA@mail.gmail.com> thusly:
There's a problem with permissions. Current library maintainer permissions are insufficient to allow Travis builds for a boostorg library. In other words: I can not flick the Travis switch for any of the Boost libraries.
Are you able to enable travis builds for your personal fork? That's what I did for a fork I have on github. In my case, the parent repository was already Travis enabled, but enabling Travis on my fork allowed me to CI build my changes before issuing a pull request. For the Travis build to be useful to the parent repo, it will of course have to be enabled there as well. -- "The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline> The Computer Graphics Museum <http://computergraphicsmuseum.org> The Terminals Wiki <http://terminals.classiccmp.org> Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>
On Thu, Sep 18, 2014 at 12:38 PM, Antony Polukhin <antoshkka@gmail.com> wrote:
A few things make me nervous.
First of all, TravisCI requires some access permissions to the repo. I'm not a github expert so I'm not 100% sure that this is safe.
You mean push permissions? Hm, looks very insecure, that would be a big concern for me, especially if it can push to boostorg repos. Why does it need push rights?
The questions.
Do we need a license note in .travis.yml file?
I suppose, all content should be licensed to avoid any confusion.
there is no policy in Boost about usage of foreign services like TravisCI and Coveralls.
Is is OK to commit to Boost repos .travis.yml files without enabling TravisCI runs? Can we enable automated testing using TravisCI for a Boost repo? Can we enable automated tests coverage using Coverall for a Boost repo?
I think you can do that with your library. My main concern is that push access should be restricted to the parties who is responsible for the code and not third party systems, which security is questionable.
2014-09-18 13:19 GMT+04:00 Andrey Semashev <andrey.semashev@gmail.com>:
On Thu, Sep 18, 2014 at 12:38 PM, Antony Polukhin <antoshkka@gmail.com> wrote:
A few things make me nervous.
First of all, TravisCI requires some access permissions to the repo. I'm not a github expert so I'm not 100% sure that this is safe.
You mean push permissions? Hm, looks very insecure, that would be a big concern for me, especially if it can push to boostorg repos. Why does it need push rights?
It does not require push rights: http://docs.travis-ci.com/user/github-oauth-scopes/ But it requires some `write:repo_hook` and `repo_deployment`. I'm not sure what does it mean. -- Best regards, Antony Polukhin
On Thu, Sep 18, 2014 at 2:42 AM, Antony Polukhin <antoshkka@gmail.com> wrote:
2014-09-18 13:19 GMT+04:00 Andrey Semashev <andrey.semashev@gmail.com>:
On Thu, Sep 18, 2014 at 12:38 PM, Antony Polukhin <antoshkka@gmail.com> wrote:
A few things make me nervous.
First of all, TravisCI requires some access permissions to the repo. I'm not a github expert so I'm not 100% sure that this is safe.
You mean push permissions? Hm, looks very insecure, that would be a big concern for me, especially if it can push to boostorg repos. Why does it need push rights?
It does not require push rights: http://docs.travis-ci.com/user/github-oauth-scopes/
But it requires some `write:repo_hook` and `repo_deployment`. I'm not sure what does it mean.
Correct, Travis-CI does not require write access to the git repo and will never push or modify commits. The hooks enable GitHub to notify Travis-CI when a pull-request is made so that Travis-CI can start a built automatically and also allow it to update the commit status on GitHub to indicate if a commit "broke the build". See the documentation at [1] for explanations of the permissions required by Travis-CI. -kyle [1] http://docs.travis-ci.com/user/github-oauth-scopes/
participants (5)
-
Andrey Semashev -
Antony Polukhin -
Kyle Lutz -
legalize+jeeves＠mail.xmission.com -
Niall Douglas