Travis/Appveyor improvements for the superproject
Travis and Appveyor tests for the Boost superproject have been improved to automatically run the tests for the library mentioned in the commit message. For example, if the message is "Update iterator from develop", the corresponding CI job runs the tests of the Iterator library (among other things). This is a bit redundant if the library itself has Travis/Appveyor set up, but keeping an eye on https://ci.appveyor.com/project/boostorg/boost/history and https://travis-ci.org/boostorg/boost/builds can't hurt. For example, Appveyor says that Iterator fails a test on msvc-12.0 (Visual Studio 2013): https://ci.appveyor.com/project/boostorg/boost/build/1.0.3744-develop and Utility fails a test on msvc-14.0/msvc-14.1: https://ci.appveyor.com/project/boostorg/boost/build/1.0.3765-develop
On 9/20/2017 9:33 AM, Peter Dimov via Boost wrote:
Travis and Appveyor tests for the Boost superproject have been improved to automatically run the tests for the library mentioned in the commit message. For example, if the message is "Update iterator from develop", the corresponding CI job runs the tests of the Iterator library (among other things).
When does the superproject automatically run tests for an individual library ? When the superproject itself gets updated, or when any given submodule gets updated ? I suspect the former.
This is a bit redundant if the library itself has Travis/Appveyor set up, but keeping an eye on
https://ci.appveyor.com/project/boostorg/boost/history
and
https://travis-ci.org/boostorg/boost/builds
can't hurt.
For example, Appveyor says that Iterator fails a test on msvc-12.0 (Visual Studio 2013):
https://ci.appveyor.com/project/boostorg/boost/build/1.0.3744-develop
and Utility fails a test on msvc-14.0/msvc-14.1:
https://ci.appveyor.com/project/boostorg/boost/build/1.0.3765-develop
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Edward Diener wrote:
On 9/20/2017 9:33 AM, Peter Dimov via Boost wrote:
Travis and Appveyor tests for the Boost superproject have been improved to automatically run the tests for the library mentioned in the commit message. For example, if the message is "Update iterator from develop", the corresponding CI job runs the tests of the Iterator library (among other things).
When does the superproject automatically run tests for an individual library ? When the superproject itself gets updated, or when any given submodule gets updated ? I suspect the former.
There is a script that, when a submodule repo is updated, updates the superproject to point at the new submodule commit. https://github.com/boostorg/boost/commits/develop All those commits by "boost-commitbot" are done by this script. The CI scripts that are invoked per superproject commit parse the commit message "Update atomic from develop" and run the tests of the updated submodule (Atomic in this case): https://ci.appveyor.com/project/boostorg/boost/build/1.0.3771-develop
On 20/09/2017 16:20, Peter Dimov via Boost wrote:
Edward Diener wrote:
Travis and Appveyor tests for the Boost superproject have been improved > to automatically run the tests for the library mentioned in
On 9/20/2017 9:33 AM, Peter Dimov via Boost wrote: the commit > message. For example, if the message is "Update iterator from develop", > the corresponding CI job runs the tests of the Iterator library (among > other things).
When does the superproject automatically run tests for an individual library ? When the superproject itself gets updated, or when any given submodule gets updated ? I suspect the former.
There is a script that, when a submodule repo is updated, updates the superproject to point at the new submodule commit.
https://github.com/boostorg/boost/commits/develop
All those commits by "boost-commitbot" are done by this script.
The CI scripts that are invoked per superproject commit parse the commit message "Update atomic from develop" and run the tests of the updated submodule (Atomic in this case):
https://ci.appveyor.com/project/boostorg/boost/build/1.0.3771-develop
Shouldn't the hole boost repo be tested when a library changes? bjam should just run tests that only depend on those headers. Otherwise, a library could just break several libraries but we would not catch them until the dependent library is updated. Or this "test whole boost" tasks is delegated to the regression testers? Best, Ion
Ion Gaztañaga wrote:
Shouldn't the hole boost repo be tested when a library changes?
This isn't feasible as CI jobs have a time limit of 49 minutes on Travis, which is sometimes insufficient for testing even a single library. I did add a "quick" test to status/Jamfile that runs the "quick" target of each test Jamfile, but that needs to complete within a few seconds, so you can't do much beyond simple smoke checks there.
On 20/09/2017 15:45, Peter Dimov via Boost wrote:
Ion Gaztañaga wrote:
Shouldn't the hole boost repo be tested when a library changes?
This isn't feasible as CI jobs have a time limit of 49 minutes on Travis, which is sometimes insufficient for testing even a single library.
I think this is the problem with this approach: any non-trivial library will quickly run out of time on the CI servers, some of which are rather slow. Certainly Math and Multiprecision libraries had to be split into multiple jobs to get through CI testing. I'm sure there are many others with similar issues. Best, John. --- This email has been checked for viruses by AVG. http://www.avg.com
This isn't feasible as CI jobs have a time limit of 49 minutes on Travis, which is sometimes insufficient for testing even a single library.
I think this is the problem with this approach: any non-trivial library will quickly run out of time on the CI servers, some of which are rather slow. Certainly Math and Multiprecision libraries had to be split into multiple jobs to get through CI testing. I'm sure there are many others with similar issues.
There is another related issue: unless everyone is completely disciplined about putting [CI skip] in commit messages for documentation only commits, this is going to trigger a lot of needless jobs? Best, John. --- This email has been checked for viruses by AVG. http://www.avg.com
On 20 September 2017 at 18:52, John Maddock via Boost
This isn't feasible as CI jobs have a time limit of 49 minutes on Travis, which is sometimes insufficient for testing even a single library.
I think this is the problem with this approach: any non-trivial library will quickly run out of time on the CI servers, some of which are rather slow. Certainly Math and Multiprecision libraries had to be split into multiple jobs to get through CI testing. I'm sure there are many others with similar issues.
There is another related issue: unless everyone is completely disciplined about putting [CI skip] in commit messages for documentation only commits, this is going to trigger a lot of needless jobs?
FYI, AppVeyor allows to configure commit filters to only build if C++ files or Jamfiles are touched, for instance. https://www.appveyor.com/docs/how-to/filtering-commits/ Best regards, -- Mateusz Loskot, http://mateusz.loskot.net
On 20 September 2017 at 17:52, John Maddock via Boost
There is another related issue: unless everyone is completely disciplined about putting [CI skip] in commit messages for documentation only commits, this is going to trigger a lot of needless jobs?
[CI skip] won't work at the moment as it won't be in the super project commit message. I suppose I could change the script to check for it and it to the super project commit message if appropriate.
John Maddock wrote:
There is another related issue: unless everyone is completely disciplined about putting [CI skip] in commit messages for documentation only commits, this is going to trigger a lot of needless jobs?
That's not a problem, the superproject jobs are not a bottleneck.
On 09/20/17 16:33, Peter Dimov via Boost wrote:
Travis and Appveyor tests for the Boost superproject have been improved to automatically run the tests for the library mentioned in the commit message. For example, if the message is "Update iterator from develop", the corresponding CI job runs the tests of the Iterator library (among other things).
This is a bit redundant if the library itself has Travis/Appveyor set up, but keeping an eye on
https://ci.appveyor.com/project/boostorg/boost/history
and
https://travis-ci.org/boostorg/boost/builds
can't hurt.
For example, Appveyor says that Iterator fails a test on msvc-12.0 (Visual Studio 2013):
https://ci.appveyor.com/project/boostorg/boost/build/1.0.3744-develop
and Utility fails a test on msvc-14.0/msvc-14.1:
https://ci.appveyor.com/project/boostorg/boost/build/1.0.3765-develop
Is this why there are now lots of blank CI columns in the test matrix? http://www.boost.org/development/tests/develop/developer/detail.html I've been seeing these for a while, and they are always blank.
On Wed, Sep 20, 2017 at 11:58 AM, Andrey Semashev via Boost < boost@lists.boost.org> wrote:
Is this why there are now lots of blank CI columns in the test matrix?
No. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
Andrey Semashev wrote:
Is this why there are now lots of blank CI columns in the test matrix?
http://www.boost.org/development/tests/develop/developer/detail.html
I've been seeing these for a while, and they are always blank.
No, that's a separate thing.
For some libraries like winapi, if changes occur you want to build it and everything that depends on it (there are about 10 libraries that do) to make sure nothing got broken. On Wed, Sep 20, 2017 at 4:10 PM, Peter Dimov via Boost < boost@lists.boost.org> wrote:
Andrey Semashev wrote:
Is this why there are now lots of blank CI columns in the test matrix?
http://www.boost.org/development/tests/develop/developer/detail.html
I've been seeing these for a while, and they are always blank.
No, that's a separate thing.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman /listinfo.cgi/boost
On 22 September 2017 at 00:29, James E. King, III via Boost
For some libraries like winapi, if changes occur you want to build it and everything that depends on it (there are about 10 libraries that do) to make sure nothing got broken.
I doubt that's feasible on a free offering from a commercial service. You can still check the normal test runners which use a wider variety of platforms (obviously that doesn't apply to winapi, but useful for most other libraries), and can set up 'cron jobs' on travis so that libraries are regularly tested with the latest versions of their dependencies (maybe best done on your own account though).
On Fri, Sep 22, 2017 at 2:29 AM, James E. King, III via Boost
On Wed, Sep 20, 2017 at 4:10 PM, Peter Dimov via Boost < boost@lists.boost.org> wrote:
Andrey Semashev wrote:
Is this why there are now lots of blank CI columns in the test matrix?
http://www.boost.org/development/tests/develop/developer/detail.html
I've been seeing these for a while, and they are always blank.
No, that's a separate thing.
For some libraries like winapi, if changes occur you want to build it and everything that depends on it (there are about 10 libraries that do) to make sure nothing got broken.
No argument about that. But that doesn't mean there have to be blank columns. BTW, we do run tests on all libraries periodically, so you can see how other libraries react to your changes. The downside is that you don't get notifications. PS: Please do not top post.
On 20 September 2017 at 14:33, Peter Dimov via Boost
Travis and Appveyor tests for the Boost superproject have been improved to automatically run the tests for the library mentioned in the commit message. For example, if the message is "Update iterator from develop", the corresponding CI job runs the tests of the Iterator library (among other things).
This seems like a bad idea. There are already modules it won't work for, and it won't work for manual commits, for which testing is probably more important. It would be better to get the paths from git than scraping it from a commit message. You should already have a checkout of the super project, so something like this should work: import subprocess, re repo_path = '.' commit = 'HEAD' raw_diff = subprocess.check_output(['git', '-C', repo_path, 'diff', commit + '^', commit, '--raw']) paths = re.findall(r'^:\d{6} 160000 [a-z0-9.]+ [a-z0-9.]+ [A-Z]\t(.*)$', raw_diff, re.M) print paths
participants (9)
-
Andrey Semashev -
Daniel James -
Edward Diener -
Ion Gaztañaga -
James E. King, III -
John Maddock -
Mateusz Loskot -
Peter Dimov -
Rene Rivera