Hi there, with the 1.34.1 release out the door, what are the plans concerning the repository and the various branches ? Where should bug fixes and other patches go ? Is the release branch going to become the new development branch, as was suggested in a mail some moons ago ? Should I wait for the subversion migration before doing anything ? What is the timeframe for the migration ? Thanks for clarifying any of the above points. Regards, Stefan -- ...ich hab' noch einen Koffer in Berlin...
Stefan Seefeld wrote:
with the 1.34.1 release out the door, what are the plans concerning the repository and the various branches ? Where should bug fixes and other patches go ? Is the release branch going to become the new development branch, as was suggested in a mail some moons ago ?
The Boost 1.34.1 release branch will certainly be the basis of Boost 1.35.0. I'm not certain we've figured out quite how to handle merging the bits from trunk over to that branch. The good news is that Subversion makes it very, very easy to move directories around.
Should I wait for the subversion migration before doing anything ?
No. There's a lot on the CVS trunk that will need to be merged over anyway; so there's no reason to hold off on doing useful work.
What is the timeframe for the migration ?
I'll send out an announcement when we're sure, but I'd like to move over early next week. We'll give a couple days notice before we switch, of course, so that everyone can get any outstanding work committed to CVS. - Doug
Douglas Gregor wrote:
The Boost 1.34.1 release branch will certainly be the basis of Boost 1.35.0. I'm not certain we've figured out quite how to handle merging the bits from trunk over to that branch. The good news is that Subversion makes it very, very easy to move directories around.
OK. For avoidance of doubt, that is the "RC_1_34_0" branch, right ?
What is the timeframe for the migration ?
I'll send out an announcement when we're sure, but I'd like to move over early next week. We'll give a couple days notice before we switch, of course, so that everyone can get any outstanding work committed to CVS.
That's great news. Thanks ! Stefan -- ...ich hab' noch einen Koffer in Berlin...
Stefan Seefeld wrote:
Douglas Gregor wrote:
The Boost 1.34.1 release branch will certainly be the basis of Boost 1.35.0. I'm not certain we've figured out quite how to handle merging the bits from trunk over to that branch. The good news is that Subversion makes it very, very easy to move directories around.
OK. For avoidance of doubt, that is the "RC_1_34_0" branch, right ?
Yes. - Doug
On Thursday 26 July 2007 08:58, Douglas Gregor wrote:
What is the timeframe for the migration ?
I'll send out an announcement when we're sure, but I'd like to move over early next week. We'll give a couple days notice before we switch, of course, so that everyone can get any outstanding work committed to CVS.
I know this is rather late in the game but it's only recently that I've had extensive experience with Subversion. My experience has not been good. I've found that Subversion scales very poorly, which makes me nervous about moving a large project like Boost into it. I've had situations where's it's taken hours and sometimes days to perform checkouts. And this is with the most recent version. I've also encountered issues with Subversion's interaction with http timeouts. Apparently Subversion sets timeouts low enough that they can expire when operating on working directories that exist on networked filesystems. It can also happen when checking out or updating very large files. These timeouts are considered a "feature" by the Subversion folks. See: http://lists.cs.uiuc.edu/pipermail/llvmdev/2007-July/010066.html Read the entire thread and the links in it for the whole story. The bottom line is that svn over http and https is very, very broken. (My comment about no problems with other svn repositories probably has to do with the fact that those other repositories are much smaller than llvm's). These timeout errors have on occasion resulted in corrupt working directories where an svn cleanup doesn't fix the problem. More than once I've had to blow away a working directory and start over. Fortunately, I haven't yet had to do this on a directory with pending changes, but I'm sure it's only a matter of time. I'm now VERY careful about when and how I do updates. More careful than I should have to be. I don't expect Boost to suddenly change course and abandon the plans to more to Subversion, but I would ask that the community be open to changing to something else if these problems arise. Maybe they will, maybe they won't, but the fact that I've had so many problems and that those problems have impacted critical work makes me not optimistic. If we were back at the beginning of discussions about what SCM to use, I would now strongly, highly, exclusively recommend git. It's been fabulous for some other projects I'm working on. Again, this is just a heads up. I don't intend to re-open the debate. But please be open to suggestions for change if problems crop up. -Dave
On Jul 31, 2007, at 12:29 PM, David A. Greene wrote:
I don't expect Boost to suddenly change course and abandon the plans to more to Subversion, but I would ask that the community be open to changing to something else if these problems arise. Maybe they will, maybe they won't, but the fact that I've had so many problems and that those problems have impacted critical work makes me not optimistic.
Frankly, I don't think this is productive. Of course we're open to suggestions should problems arise, but the fact is that many software projects far larger than Boost have successfully used Subversion and we have no reason to expect problems. - Doug
David A. Greene wrote:
I know this is rather late in the game but it's only recently that I've had extensive experience with Subversion.
My experience has not been good. I've found that Subversion scales very poorly, which makes me nervous about moving a large project like Boost into it. I've had situations where's it's taken hours and sometimes days to perform checkouts. And this is with the most recent version. ..... I've also encountered issues with Subversion's interaction with http timeouts. ..... The bottom line is that svn over http and https is very, very broken.
(My comment about no problems with other svn repositories probably has to do with the fact that those other repositories are much smaller than llvm's).
If I understand correctly, no other LLVM developer has this problem, right? Then, it suggests either some particular flakiness in your network environment, or your svn client is somehow broken. And there are many projects larger than LLVM that don't have any problems. - Volodya
On 7/31/07, Vladimir Prus <ghost@cs.msu.su> wrote:
David A. Greene wrote:
I know this is rather late in the game but it's only recently that I've had extensive experience with Subversion.
My experience has not been good. I've found that Subversion scales very poorly, which makes me nervous about moving a large project like Boost into it. I've had situations where's it's taken hours and sometimes days to perform checkouts. And this is with the most recent version. ..... I've also encountered issues with Subversion's interaction with http timeouts. ..... The bottom line is that svn over http and https is very, very broken.
(My comment about no problems with other svn repositories probably has to do with the fact that those other repositories are much smaller than llvm's).
If I understand correctly, no other LLVM developer has this problem, right? Then, it suggests either some particular flakiness in your network environment, or your svn client is somehow broken. And there are many projects larger than LLVM that don't have any problems.
People regularly have similar problems with my toy project subversion server, which might indicate that the issue is not a function of project size (but rather of bad administration on my side? ;-). Regards, Michael
On Tuesday 31 July 2007 13:47, Vladimir Prus wrote:
If I understand correctly, no other LLVM developer has this problem, right?
No one else has reported it.
Then, it suggests either some particular flakiness in your network environment, or your svn client is somehow broken.
If "flakiness" means "latency," then that may very well be the case. But network latency is not an issue of "flake." It's a reality of life. Things should not break due to latency. Certainly working directories should not get corrupted! That is a showstopper for me and is why I refuse to put any of my personal projects into Subversion.
And there are many projects larger than LLVM that don't have any problems.
So maybe it's not a scale issue. All I know is that I've had real problems with Subversion, even after upgrading to the latest release. Others have reported similar problems on other projects. So even if no other _llvm_ developer has seen these problems, other developers on other projects have. I'm just giving the Boost community a heads-up as to my personal experience. The community is free to decide whether it's useful information. I don't use the Boost repository all that much and I don't have commit access so it makes no difference to me. -Dave
On Jul 31, 2007, at 3:40 PM, David A. Greene wrote:
Then, it suggests either some particular flakiness in your network environment, or your svn client is somehow broken.
If "flakiness" means "latency," then that may very well be the case. But network latency is not an issue of "flake." It's a reality of life. Things should not break due to latency. Certainly working directories should not get corrupted! That is a showstopper for me and is why I refuse to put any of my personal projects into Subversion.
Okay, I looked into this. Here's my theory: you are checking out a large repository to a networked filesystem, say, NFS. The Subversion client downloads a ton of data and writes it to many, many small files. The networked filesystem slows to a crawl under the load (creating many small files is a particularly bad case for many networked file systems), and essentially the Subversion client can't keep up with the server that is feeding its data. After a while, the Subversion server gets bored of waiting for the client and closes the connection. Why does turning off http compression help? Probably because turning off compression makes for much more Subversion server->client traffic, and gives the networked file system time to catch up when it is writing files. Since compression ratios for source code can be very high, and networked file systems generally don't compress data when transmitting files, there would be a very large imbalance between the server->client transfers and the client->file system transfers. What's the solution? If I'm correct, checking out to a local file system should solve the problem immediately (and would be much, much faster anyway). Additionally, we can (and have) bump up the timeout values on the server side to try to keep it from severing connections too early. Anyway, try out the Boost Subversion repository to see if it works for you. If not, we'll see what we can do. - Doug
On Jul 31, 2007, at 7:08 PM, Douglas Gregor wrote:
Here's my theory: you are checking out a large repository to a networked filesystem, say, NFS. The Subversion client downloads a ton of data and writes it to many, many small files. The networked filesystem slows to a crawl under the load (creating many small files is a particularly bad case for many networked file systems), and essentially the Subversion client can't keep up with the server that is feeding its data. After a while, the Subversion server gets bored of waiting for the client and closes the connection.
More evidence for this theory... you seem to have said that upgrading from Subversion 1.3.1 to 1.4.x helped a bit. Since 1.4.0 introduced a new, more compact representation of the working copy of a Subversion repository, a checkout with 1.4.x requires much less disk activity (= network activity, for a networked file system) than a checkout with pre-1.4.0 Subversion. So, you wouldn't see the timeout problem as often, because the networked file system is more likely to keep up with the server. - Doug
On Tuesday 31 July 2007 18:12, Douglas Gregor wrote:
On Jul 31, 2007, at 7:08 PM, Douglas Gregor wrote:
Here's my theory: you are checking out a large repository to a networked filesystem, say, NFS. The Subversion client downloads a ton of data and writes it to many, many small files. The networked filesystem slows to a crawl under the load (creating many small files is a particularly bad case for many networked file systems), and essentially the Subversion client can't keep up with the server that is feeding its data. After a while, the Subversion server gets bored of waiting for the client and closes the connection.
I absolutely agree with your diagnosis. The problem with Subversion is not the timeouts per se. If that's all that happened, it would be an annoyance. The problem with Subversion is that these timeouts can sometimes cause working directory corruption. That is absolutely, positively unacceptable for anyone doing critical work. Note that I'm not talking about errors from the network filesystem itself (broken connections causing errors on file I/O, etc.). I'm talking about http protocol timeouts. Subversion should be resilient enough to exit gracefully under such circumstances and it doesn't always do that. -Dave
2007/8/1, David A. Greene <greened@obbligato.org>:
On Tuesday 31 July 2007 18:12, Douglas Gregor wrote:
On Jul 31, 2007, at 7:08 PM, Douglas Gregor wrote:
Here's my theory: you are checking out a large repository to a networked filesystem, say, NFS. The Subversion client downloads a ton of data and writes it to many, many small files. The networked filesystem slows to a crawl under the load (creating many small files is a particularly bad case for many networked file systems), and essentially the Subversion client can't keep up with the server that is feeding its data. After a while, the Subversion server gets bored of waiting for the client and closes the connection.
I absolutely agree with your diagnosis.
The problem with Subversion is not the timeouts per se. If that's all that happened, it would be an annoyance.
The problem with Subversion is that these timeouts can sometimes cause working directory corruption. That is absolutely, positively unacceptable for anyone doing critical work. Note that I'm not talking about errors from the network filesystem itself (broken connections causing errors on file I/O, etc.). I'm talking about http protocol timeouts. Subversion should be resilient enough to exit gracefully under such circumstances and it doesn't always do that.
The FAQ mentions some pitfalls: http://subversion.tigris.org/faq.html#nfs /$
2007/8/1, David A. Greene <greened@obbligato.org>:
On Tuesday 31 July 2007 18:12, Douglas Gregor wrote:
On Jul 31, 2007, at 7:08 PM, Douglas Gregor wrote:
Here's my theory: you are checking out a large repository to a networked filesystem, say, NFS. The Subversion client downloads a ton of data and writes it to many, many small files. The networked filesystem slows to a crawl under the load (creating many small files is a particularly bad case for many networked file systems), and essentially the Subversion client can't keep up with the server that is feeding its data. After a while, the Subversion server gets bored of waiting for the client and closes the connection.
I absolutely agree with your diagnosis.
The problem with Subversion is not the timeouts per se. If that's all that happened, it would be an annoyance.
The problem with Subversion is that these timeouts can sometimes cause working directory corruption. That is absolutely, positively unacceptable for anyone doing critical work. Note that I'm not talking about errors from the network filesystem itself (broken connections causing errors on file I/O, etc.). I'm talking about http protocol timeouts. Subversion should be resilient enough to exit gracefully under such circumstances and it doesn't always do that.
I have had some experience of svn HTTP timeouts resulting from an overloaded server (nothing to do with Boost). In this case, it looked as if the working copy had become corrupt. However, after asking on the svn mailing list, it turned out that the "corruption" could be fixed with a simple command. The problem is that the client gives unhelpful error messages, and the commands to recover are unintuitive - both problems that ought to be fixable. Specifically, it seems that if an "svn commit" fails due to an HTTP timeout after the commit has happened in the repository, then you need to do an "svn update" to get the working copy back in sync, yet the error (unhelpfully) suggests "svn cleanup". Or something like that. Of course there may be other scenarios in which the corruption is real [e.g. your timeout during update rather than commit], but based on what I've seen I would never abandon an apparently-corrupt working copy without first posting a "here's what I did and here are the errors I got" question to subversion_users. Concerning subversion performance in general, I have found that the Apache authentication method chosen has a greater effect than anything else. Regards, Phil.
Stefan Seefeld wrote:
with the 1.34.1 release out the door, what are the plans concerning the repository and the various branches ? Where should bug fixes and other patches go ? Is the release branch going to become the new development branch, as was suggested in a mail some moons ago ?
As Doug said in his reply, "The Boost 1.34.1 release branch will certainly be the basis of Boost 1.35.0." The mechanism for that isn't for 1.34.1 to become the new development branch. Rather, development occurs in "trunk" as always. 1.34.1 becomes the new "stable" branch. As libraries become ready for 1.35.0, they get merged into "stable". The criteria for being ready is that they don't destabilize the "stable" branch. Thus "stable" is always in a state that would allow it to be shipped with relatively little effort. At least that's the vision I'm putting forward. We don't have the exact mechanisms worked out. There is a lot of concern that we not introduce complex new tools, because of concern over manpower and fragility. I've been trying out procedures based on the current bjam, regression testing, and Subversion tools. The first step in all of this was to get 1.34.1 out the door. That's done, thanks to Thomas Witt and other's hard work. The second step is to get changed over to Subversion. That's well underway, led by Doug Gregor. We'll start planning the rest of it in the new few weeks. --Beman
participants (10)
-
Beman Dawes -
David A. Greene -
Doug Gregor -
Douglas Gregor
-
Douglas Gregor -
Henrik Sundberg -
Michael Walter -
Phil Endecott -
Stefan Seefeld -
Vladimir Prus