Last summer we had a discussion about Travis CI Resource Usage for boostorg. Originally the discussion was about purchasing more resources; however the discussion quickly focused on finding the root cause of the backlog. The desire is to ensure that pull requests submitted to repositories are processed in a timely manner. This is not happening, and there is at least a 12 hour backlog on pull request builds for boostorg right now. Summarizing the discussion back then, it was proposed that the backlog was caused by folks working on feature branches directly in their boostorg repository, with the Travis CI setting enabled to "build branch updates". The proposed solution was to encourage everybody to fork their repository and set up their own Travis CI and Appveyor under their account, and work feature branches on your fork. If working alone, this is efficient for everyone as we all end up with faster builds because your account will not contend with boostorg for resources. So please, use your personal account to build up features if you can. Additionally, tuning your Travis CI build script to do a little bit less work will help. If you have a script that builds with gcc, gcc-5, gcc-6, gcc-7, clang, and cxxstd of 03,11,14,1z - for most libraries that are at the c++03 language level this is going to be significant overkill for use as a pull request gate. In the example above, there are 20 builds to do; double that if you build debug and release variants. If each build takes just 2 minutes that's still an 80 minute CI cycle. Please spend some time reviewing your practices around the use of the free CI resources we all share. Thanks, Jim
James E. King, III wrote:
The desire is to ensure that pull requests submitted to repositories are processed in a timely manner. This is not happening, and there is at least a 12 hour backlog on pull request builds for boostorg right now.
This is caused by Travis Mac resources being insufficient. There is a multiple day backlog for Mac jobs, and it doesn't seem to be going away - Travis increased their job count from 180 to 204 temporarily for the weekend, it didn't help, the backlog is at the moment at 1100+ and keeps growing. Look at https://www.traviscistatus.com/ for the stats. This should in principle not affect the Linux-only .travis.yml configurations, but it does, because the queued Mac jobs result in only 3 parallel jobs being available for Linux builds, instead of the usual 5. I'd say that boostorg/math might scale down its Mac jobs a bit. :-) If this persists, we should probably not use Mac jobs at all, although they are kind of essential for Config and Build. Most other repos have a single Mac job, with a few exceptions.
On Mon, Jan 22, 2018 at 9:05 AM, Peter Dimov via Boost < boost@lists.boost.org> wrote:
James E. King, III wrote:
The desire is to ensure that pull requests submitted to repositories are
processed in a timely manner. This is not happening, and there is at least a 12 hour backlog on pull request builds for boostorg right now.
This is caused by Travis Mac resources being insufficient. There is a multiple day backlog for Mac jobs, and it doesn't seem to be going away - Travis increased their job count from 180 to 204 temporarily for the weekend, it didn't help, the backlog is at the moment at 1100+ and keeps growing. Look at https://www.traviscistatus.com/ for the stats.
This should in principle not affect the Linux-only .travis.yml configurations, but it does, because the queued Mac jobs result in only 3 parallel jobs being available for Linux builds, instead of the usual 5.
I'd say that boostorg/math might scale down its Mac jobs a bit. :-)
If this persists, we should probably not use Mac jobs at all, although they are kind of essential for Config and Build. Most other repos have a single Mac job, with a few exceptions.
I'm not convinced it is due to mac jobs. I have no mac jobs in three repositories because of the delays there; my linux-only PRs are still experiencing a 12 hour delay running. Right now I am supposed to submit PRs for review, so I'm not sure I have much of a choice but to wait for the 12 hour cycle. I have been running builds in my fork to get them working and then submitting the PR up to the boostorg repo to get the public code review. Travis is at least smart enough not to block the 5 available build runners if there are no mac resources to run anything... it'll keep running 5 concurrent. One problem is we all have to share 5 concurrent. That can be fixed with money. I just sync'd with the top level develop this morning and saw a number of feature branches in boostorg repositories were added, so I don't think the original guidance from last summer was heeded. - Jim
James E. King, III wrote:
I'm not convinced it is due to mac jobs. I have no mac jobs in three repositories because of the delays there; my linux-only PRs are still experiencing a 12 hour delay running.
As I said, stuck Mac jobs reduce the Linux parallelism from 5 to 3.
Travis is at least smart enough not to block the 5 available build runners if there are no mac resources to run anything... it'll keep running 5 concurrent.
No, it doesn't, it runs 3. Look at it. Your delay at this specific moment is indeed due to boostorg/system feature branch work, but that's because you're lucky. Usually, you'll be stuck waiting for something like https://travis-ci.org/boostorg/math/builds/330892193 and no amount of rallying against feature branches will save you.
On Mon, Jan 22, 2018 at 9:33 AM, Peter Dimov via Boost < boost@lists.boost.org> wrote:
James E. King, III wrote:
I'm not convinced it is due to mac jobs. I have no mac jobs in three
repositories because of the delays there; my linux-only PRs are still experiencing a 12 hour delay running.
As I said, stuck Mac jobs reduce the Linux parallelism from 5 to 3.
Travis is at least smart enough not to block the 5 available build runners
if there are no mac resources to run anything... it'll keep running 5 concurrent.
No, it doesn't, it runs 3. Look at it.
Your delay at this specific moment is indeed due to boostorg/system feature branch work, but that's because you're lucky. Usually, you'll be stuck waiting for something like https://travis-ci.org/boostorg /math/builds/330892193 and no amount of rallying against feature branches will save you.
Have the issues in repositories abusing the build job count and feature branch-on-boostorg been given any sort of Github issue, trac report, or official warning? Do we know all the offenders? This is really bad behavior that affects all of us and it needs to be remedied. These builds could be done on a personal fork through Travis and Appveyor just as easily. - Jim
Usually, you'll be stuck waiting for something like https://travis-ci.org/boostorg/math/builds/330892193 and no amount of rallying against feature branches will save you.
That said, you're quite correct that working from a fork is a superior workflow.
On 01/22/18 18:13, Peter Dimov via Boost wrote:
Usually, you'll be stuck waiting for something like https://travis-ci.org/boostorg/math/builds/330892193 and no amount of rallying against feature branches will save you.
That said, you're quite correct that working from a fork is a superior workflow.
It's inconvenient though. I work with Boost repos exclusively on boostorg (except when I prepare PRs for repos I have no push permissions for), so I always know where the most recent version is. Having to select the remote whenever you push changes adds more churn and another layer of "need to merge" reminders in my backlog. Also, I think multiple remotes don't work with "git push --all". IMHO, if Travis doesn't work for me or the community on boostorg, I will probably consider disabling it altogether in my repos.
Andrey Semashev wrote:
That said, you're quite correct that working from a fork is a superior workflow.
It's inconvenient though.
It's pretty inconvenient as making a Boost library work out-of-superproject can be a bit of a challenge, and switching the in-superproject directory to the fork has its downsides too. On the other hand though, Travis turnaround improves drastically as you're back to 5 jobs and don't wait for anyone else, and people can no longer blame you for their boostorg jobs being stuck. So it seems worth looking into. All that said, we (boostorg) could really use 10 jobs, and whatever better Mac support money can buy.
On Mon, Jan 22, 2018 at 10:14 AM, Peter Dimov via Boost < boost@lists.boost.org> wrote:
All that said, we (boostorg) could really use 10 jobs, and whatever better Mac support money can buy.
AFAIK the key blocker for getting more resources on Travis is the pay structure they have. In order to get a better plan than the free OSS plan you'd have to get their top tier paid plan. It is my understanding that the Travis infrastructure setup is not capable of doing mixed plans which would be the ideal for OSS as it would allow just adding more resources on top of the free allotment. AFAIK Travis is working to remedy the situation though. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net
On 01/22/18 17:05, Peter Dimov via Boost wrote:
James E. King, III wrote:
The desire is to ensure that pull requests submitted to repositories are processed in a timely manner. This is not happening, and there is at least a 12 hour backlog on pull request builds for boostorg right now.
This is caused by Travis Mac resources being insufficient. There is a multiple day backlog for Mac jobs, and it doesn't seem to be going away - Travis increased their job count from 180 to 204 temporarily for the weekend, it didn't help, the backlog is at the moment at 1100+ and keeps growing. Look at https://www.traviscistatus.com/ for the stats.
This should in principle not affect the Linux-only .travis.yml configurations, but it does, because the queued Mac jobs result in only 3 parallel jobs being available for Linux builds, instead of the usual 5.
I'd say that boostorg/math might scale down its Mac jobs a bit. :-)
If this persists, we should probably not use Mac jobs at all, although they are kind of essential for Config and Build. Most other repos have a single Mac job, with a few exceptions.
Are more OS X jobs available on paid subscriptions?
Andrey Semashev wrote:
Are more OS X jobs available on paid subscriptions?
Paid subscriptions use a separate OS X pool, so presumably, yes. They don't appear on the public status page though, so I don't know how things stand with certainty. For Linux turnaround, there's also the option of paying for "premium open source" with 10 parallel jobs.
On 22/01/2018 14:05, Peter Dimov via Boost wrote:
James E. King, III wrote:
The desire is to ensure that pull requests submitted to repositories are processed in a timely manner. This is not happening, and there is at least a 12 hour backlog on pull request builds for boostorg right now.
This is caused by Travis Mac resources being insufficient. There is a multiple day backlog for Mac jobs, and it doesn't seem to be going away - Travis increased their job count from 180 to 204 temporarily for the weekend, it didn't help, the backlog is at the moment at 1100+ and keeps growing. Look at https://www.traviscistatus.com/ for the stats.
This should in principle not affect the Linux-only .travis.yml configurations, but it does, because the queued Mac jobs result in only 3 parallel jobs being available for Linux builds, instead of the usual 5.
I'd say that boostorg/math might scale down its Mac jobs a bit. :-)
Done. We've had a lot of churn in PR's lately, and it's generated a whole slew of CI jobs. John. --- This email has been checked for viruses by AVG. http://www.avg.com
participants (5)
-
Andrey Semashev -
James E. King, III -
John Maddock -
Peter Dimov -
Rene Rivera