[git] [conversion] Schedule and remaining showstoppers?

Beman Dawes

15 Oct 2013 15 Oct '13

12:48 p.m.

It seems to me the best time for the svn to modular git changeover is right after a Boost release. That will give us the maximum time window to get the new house in order before the subsequent release. So should we convert right after 1.55 ships? I'm well aware that there is a long list of conversion related tasks that we haven't completed. But the svn -> git conversion process itself is so complex that only Dave and Daniel can make modifications. They have done a staggering amount of work already and are nearing exhaustion. Once the conversion is done, there are many of us that understand git well enough to help with problems. For example, https://github.com/ryppl/Boost2Git/issues/5, "Mimic gitfow", is a showstopper in that what is now "master" should be "develop" and the true "master" (i.e. releases) branch needs to be synthesized. But while it would be great if the Boost2Git process could handle that, it could probably also be do by a post-conversion step performed by git and other command line tools that many of us could help with. Messy, but I'm not willing to take a further delay waiting for perfection. Which of the issues in https://github.com/ryppl/Boost2Git/issues?state=openare true show stoppers? Are there any true showstoppers missing from that list? --Beman

Show replies by date

Niall Douglas

15 Oct 15 Oct

3:37 p.m.

On 15 Oct 2013 at 8:48, Beman Dawes wrote:

...

It seems to me the best time for the svn to modular git changeover is right after a Boost release. That will give us the maximum time window to get the new house in order before the subsequent release. So should we convert right after 1.55 ships?

I agree about switching just after a release, but it's not ready for primetime yet IMHO.

...

I'm well aware that there is a long list of conversion related tasks that we haven't completed. But the svn -> git conversion process itself is so complex that only Dave and Daniel can make modifications. They have done a staggering amount of work already and are nearing exhaustion. Once the conversion is done, there are many of us that understand git well enough to help with problems.

Firstly, I think most on this list are unaware just how hideously difficult Boost2Git has become. Every time you think you're getting closer to done, another new minefield of subtle issues appears which causes more borkage across the whole conversion process. Dave and Daniel absolutely deserve a huge round of applause for wrestling that bull out of its china shop.

...

Are there any true showstoppers missing from that list?

I would really *strongly* advocate a period where the SVN repo is used for write only, but the git repos are used for everything except writes. Boost2Git has been designed exactly to do that. We should take advantage of it. Such is the need for proper testing I would also expect such a test period to last at least two Boost releases. More, if people want all of sandbox working perfectly. I also think this list's opinion on Boost2Git's readiness is insignificant compared to Dave's and Daniel's. I only messed with the conversion code for a short while, and I came away feeling awed. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

Tom Kent

4:33 p.m.

On Tue, Oct 15, 2013 at 10:37 AM, Niall Douglas wrote:

...

I would really *strongly* advocate a period where the SVN repo is used for write only, but the git repos are used for everything except writes. Boost2Git has been designed exactly to do that. We should take advantage of it.

Is this at a point where we could switch the trunk/release regression tests over to it? Tom

Niall Douglas

5:55 p.m.

On 15 Oct 2013 at 11:33, Tom Kent wrote:

...

...
I would really *strongly* advocate a period where the SVN repo is used for write only, but the git repos are used for everything except writes. Boost2Git has been designed exactly to do that. We should take advantage of it.

Is this at a point where we could switch the trunk/release regression tests over to it?

Dave and Daniel would know the answer better than I, so I'll leave it to them to answer. They may be at a stage where running such tests daily is useful, equally they may be at a stage where doing so adds to their workload at a time they don't need that. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

Beman Dawes

5:59 p.m.

On Tue, Oct 15, 2013 at 11:37 AM, Niall Douglas wrote:

...

On 15 Oct 2013 at 8:48, Beman Dawes wrote:

...
It seems to me the best time for the svn to modular git changeover is right after a Boost release. That will give us the maximum time window to get the new house in order before the subsequent release. So should we convert right after 1.55 ships?

I agree about switching just after a release, but it's not ready for primetime yet IMHO.

Is that a general concern, or are there specific issues you are worried about?

...

...
I'm well aware that there is a long list of conversion related tasks that we haven't completed. But the svn -> git conversion process itself is so complex that only Dave and Daniel can make modifications. They have done a staggering amount of work already and are nearing exhaustion. Once the conversion is done, there are many of us that understand git well enough to help with problems.

Firstly, I think most on this list are unaware just how hideously difficult Boost2Git has become. Every time you think you're getting closer to done, another new minefield of subtle issues appears which causes more borkage across the whole conversion process. Dave and Daniel absolutely deserve a huge round of applause for wrestling that bull out of its china shop.

...
Are there any true showstoppers missing from that list?

I would really *strongly* advocate a period where the SVN repo is used for write only, but the git repos are used for everything except writes. Boost2Git has been designed exactly to do that. We should take advantage of it.

How would this work? As a Boost developer how could I use the git repos for my libraries if I can't write to them? How can I use the svn repo if I can't read from it? I have a fuzzy idea of what you are suggesting, and am interested, but would need to hear specifics of how it could be made to work in practice. --Beman

Niall Douglas

6:41 p.m.

On 15 Oct 2013 at 13:59, Beman Dawes wrote:

...

...
I agree about switching just after a release, but it's not ready for primetime yet IMHO.

Is that a general concern, or are there specific issues you are worried about?

It's the hideous complexity of doing a perfect conversion, not specific issues. One way I have not previously mentioned here is rebuilding the existing SVN repo to rectify unfortunate historical commits which make perfect git conversion very hard. I know Dave is opposed to this solution, and it certainly launches yet another new tool for converting SVN to cleaner-SVN. But it might be less work overall.

...

...
I would really *strongly* advocate a period where the SVN repo is used for write only, but the git repos are used for everything except writes. Boost2Git has been designed exactly to do that. We should take advantage of it.

How would this work? As a Boost developer how could I use the git repos for my libraries if I can't write to them? How can I use the svn repo if I can't read from it?

I have a fuzzy idea of what you are suggesting, and am interested, but would need to hear specifics of how it could be made to work in practice.

Sure. I'll think aloud here, it may or may not be useful. Option 1: I'm thinking that because git is distributed, a Boost developer can merrily commit to their local git without issue. However when one goes to push to origin, a *locally installed* hook script converts the push into a diff patch, and sends that to svn instead. I'd make it locally installed rather than remotely installed because such a push converting hook script is inevitably brittle, and only those who really know what they're doing should engage. Disadvantages: Only the brave will bother which reduces number of testers. Option 2: Add the metadata to allow subgit (a tool which implements simultaneous svn and git commits) to work with the Boost2Git output. Pushes to git are therefore mirrored by subgit into svn. We'd add a post-push hook to subgit to undo any changes by subgit to the git repo, letting the daily Boost2Git cron job do the mirror instead (unless Boost2Git is upgraded to implement delta updates). Disadvantages: Would need more work to implement subgit metadata output. Need to buy a subgit licence (they may give one for free in exchange for publicity). Users may be confused if git pushes don't immediately turn up in git origin. git push times are likely to become tens of seconds. Option 3: Instead of doing an all-or-nothing SVN to Git conversion, we gradually transition individual libraries out of SVN and into Git submodules using svn:externals i.e. you wipe library X from SVN, replacing it with an svn:externals which loads a Github provided SVN translation from Git (https://help.github.com/articles/support-for-subversion-clients) for that particular library submodule. This lets maintainers test their particular library to make sure it really works as a git submodule, and "detach" its write access from SVN. Slowly, over time, we therefore transition from all-SVN write access, through some-SVN some-Git write access, to all-Git write access, with the SVN repo eventually becoming a "shell" of links to git modules and whose svn revision no longer increments. Disadvantages: someone has to manage and coordinate this process, and if finding review managers is hard now, finding git conversion managers is probably harder. Explaining it coherently to Boost devs might be hard, especially convincing of need as compared to staying with SVN. I hope this is useful, even if not fully thought through. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

Beman Dawes

9:52 p.m.

On Tue, Oct 15, 2013 at 2:41 PM, Niall Douglas wrote:

...

On 15 Oct 2013 at 13:59, Beman Dawes wrote:

...
...
I agree about switching just after a release, but it's not ready for primetime yet IMHO.

Is that a general concern, or are there specific issues you are worried about?

It's the hideous complexity of doing a perfect conversion, not specific issues.

We are not trying to do a perfect conversion. We are trying to do a reasonably good conversion and then move on. We want files to appear in the right places, we want to retain history at least for trunk and branches/release, and we want a few other things. But mostly we want to move on and put the conversion behind us. Remember that we will be doing nothing that degrades the svn repo. It will be available for years and years into the future.

...

One way I have not previously mentioned here is rebuilding the existing SVN repo to rectify unfortunate historical commits which make perfect git conversion very hard. I know Dave is opposed to this solution, and it certainly launches yet another new tool for converting SVN to cleaner-SVN. But it might be less work overall.

Less work that what? I can't see how adding an additional step to build a new svn repo and then converting that could possibly be less work that finishing off the work that has already been done, and getting on with the actual conversion. <snip> ... three possible approaches ... </snip> All I'm seeing is additional complexity and few if any benefits. It is one thing to fix any remaining glitches, but quite another to re-engineer the whole process because there is fear it might have hidden problems. --Beman

Daniel Pfeifer

16 Oct 16 Oct

8:53 a.m.

2013/10/15 Beman Dawes :

...

On Tue, Oct 15, 2013 at 2:41 PM, Niall Douglas wrote:

...
On 15 Oct 2013 at 13:59, Beman Dawes wrote:

...
...
I agree about switching just after a release, but it's not ready for primetime yet IMHO.

Is that a general concern, or are there specific issues you are worried about?

It's the hideous complexity of doing a perfect conversion, not specific issues.

We are not trying to do a perfect conversion. We are trying to do a reasonably good conversion and then move on.

I think we had a reasonably good conversion at BoostCon 2012. Our approach was rejected by the community so we started a new approach, this time aiming for perfect conversion. The new approach looked very promising at first and we made good progress. But it has become more complex than expected. It is really like Niall wrote: "Every time you think you're getting closer to done, another new minefield of subtle issues appears which causes more borkage across the whole conversion process." Here is a quote from Peter Drucker: A company should build a process that systematically looks at every product, every service, every process, every policy, every market with the question, "If we weren't doing this already, knowing what we now know, would we start it, would we go into it?" If not, how quickly can we get out? The answers to these questions are: "No" and "I don't know", respectively. cheers, Daniel

Beman Dawes

12:51 p.m.

On Wed, Oct 16, 2013 at 4:53 AM, Daniel Pfeifer wrote:

...

2013/10/15 Beman Dawes :

...
...
... We are not trying to do a perfect conversion. We are trying to do a reasonably good conversion and then move on.

I think we had a reasonably good conversion at BoostCon 2012. Our approach was rejected by the community so we started a new approach, this time aiming for perfect conversion.

The new approach looked very promising at first and we made good progress. But it has become more complex than expected. It is really like Niall wrote: "Every time you think you're getting closer to done, another new minefield of subtle issues appears which causes more borkage across the whole conversion process."

Here is a quote from Peter Drucker:

A company should build a process that systematically looks at every product, every service, every process, every policy, every market with the question, "If we weren't doing this already, knowing what we now know, would we start it, would we go into it?" If not, how quickly can we get out?

Wonderful! Yet another example of why Drucker is revered in management circles.

...

The answers to these questions are: "No" and "I don't know", respectively.

My take on "how quickly can we get out?": * Declare the conversion software done. Dave and Daniel are off the hook, although we will need their help if the conversion process stops running before the cutover. * The moment 1.55 ships, we make the SVN repo read only. As soon as the next conversion run finishes, we turn off the conversion process. * We run a process that checks out all the Boost2Git repos, does the gitflow branch renaming, and pushes those changes up to the Boost2Git repos. * At that point the bulk conversion is done. Library maintainers can start work on their individual libraries. If a problem is discovered in an individual library's repo, we help that library maintainer resolve the problem for that library rather than rerunning the entire conversion for all libraries. To be able to pull that off, a lot of preparation not involving the Boost2Git conversion process remains to be done. But we need to be clear about where we are going before worrying about exactly how we are going to get there. Thanks, --Beman

Peter Dimov

1:18 p.m.

Beman Dawes wrote:

...

* The moment 1.55 ships, we make the SVN repo read only. As soon as the next conversion run finishes, we turn off the conversion process.

What happens with the changes to trunk that haven't been merged to release?

Beman Dawes

1:38 p.m.

On Wed, Oct 16, 2013 at 9:18 AM, Peter Dimov wrote:

...

Beman Dawes wrote:

* The moment 1.55 ships, we make the SVN repo read only. As soon as the

...
next conversion run finishes, we turn off the conversion process.

What happens with the changes to trunk that haven't been merged to release?

For each library, "trunk" will have been renamed "develop", "branches/release" will have been renamed "master" So a diff of develop vs master on git will show the same changes as a diff of trunk vs branches/release on svn. IOW, any changes to trunk will be preserved as changes to develop. --Beman

Peter Dimov

1:59 p.m.

Beman Dawes wrote:

...

On Wed, Oct 16, 2013 at 9:18 AM, Peter Dimov wrote:

...
Beman Dawes wrote:

* The moment 1.55 ships, we make the SVN repo read only. As soon as the

...
next conversion run finishes, we turn off the conversion process.

What happens with the changes to trunk that haven't been merged to release?

For each library, "trunk" will have been renamed "develop", "branches/release" will have been renamed "master"

So a diff of develop vs master on git will show the same changes as a diff of trunk vs branches/release on svn.

A diff will, but diff-ing isn't our procedure now. At least it isn't mine. I keep track of revisions that need to be merged to release. [85994], for example, needs merging.

Beman Dawes

2:18 p.m.

On Wed, Oct 16, 2013 at 9:59 AM, Peter Dimov wrote:

...

Beman Dawes wrote:

...
On Wed, Oct 16, 2013 at 9:18 AM, Peter Dimov wrote:

...
Beman Dawes wrote:

* The moment 1.55 ships, we make the SVN repo read only. As soon as the

...
next conversion run finishes, we turn off the conversion process.

What happens with the changes to trunk that haven't been merged to > release?

For each library, "trunk" will have been renamed "develop", "branches/release" will have been renamed "master"

So a diff of develop vs master on git will show the same changes as a diff of trunk vs branches/release on svn.

A diff will, but diff-ing isn't our procedure now. At least it isn't mine. I keep track of revisions that need to be merged to release. [85994], for example, needs merging.

Ah! I was using diff to illustrate the state of the two branches. The actual way to do the merge is a git question. Like all merges, start by switching to the branch to be merged to - "master" in this case. Then merge from the whole "develop" branch if 85994 is the only difference. Otherwise, you need to merge just the 85994 changeset. (I'm a TortoiseGit user, so am the wrong person to ask about how to do that from the command line. But "git help merge" from the command line may well tell you enough to figure it out.) --Beman

Jonathan Wakely

4:03 p.m.

On 16 October 2013 15:18, Beman Dawes wrote:

...

Like all merges, start by switching to the branch to be merged to - "master" in this case. Then merge from the whole "develop" branch if 85994 is the only difference. Otherwise, you need to merge just the 85994 changeset. (I'm a TortoiseGit user, so am the wrong person to ask about how to do that from the command line. But "git help merge" from the command line may well tell you enough to figure it out.)

I assume 85994 is an svn revision, so to git merge it you need to know the Git SHA1 commit ID for that commit. You can find that via "git log" and looking for the relevant commit message, but you can't just merge 85994 without some extra work.

Niall Douglas

4:45 p.m.

On 16 Oct 2013 at 17:03, Jonathan Wakely wrote:

...

On 16 October 2013 15:18, Beman Dawes wrote:

...
Like all merges, start by switching to the branch to be merged to - "master" in this case. Then merge from the whole "develop" branch if 85994 is the only difference. Otherwise, you need to merge just the 85994 changeset. (I'm a TortoiseGit user, so am the wrong person to ask about how to do that from the command line. But "git help merge" from the command line may well tell you enough to figure it out.)

I assume 85994 is an svn revision, so to git merge it you need to know the Git SHA1 commit ID for that commit. You can find that via "git log" and looking for the relevant commit message, but you can't just merge 85994 without some extra work.

git lets you create arbitrary refs to arbitrary SHAs, so there is absolutely nothing stopping Boost2Git filling a directory of SHA refs named after each SVN revision e.g. refs/svnrefs/12345 => abcdef... In fact, that is exactly what subgit does: it stores each svn rev to SHA as a svn prop, and each SHA to svn rev as a custom ref. This lets subgit map each to the other transparently along with needed metadata. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

Beman Dawes

5:13 p.m.

On Wed, Oct 16, 2013 at 12:45 PM, Niall Douglas wrote:

...

On 16 Oct 2013 at 17:03, Jonathan Wakely wrote:

...
On 16 October 2013 15:18, Beman Dawes wrote:

...
Like all merges, start by switching to the branch to be merged to - "master" in this case. Then merge from the whole "develop" branch if 85994 is the only difference. Otherwise, you need to merge just the 85994 changeset. (I'm a TortoiseGit user, so am the wrong person to ask about how to do that from the command line. But "git help merge" from the command line may well tell you enough to figure it out.)

I assume 85994 is an svn revision, so to git merge it you need to know the Git SHA1 commit ID for that commit. You can find that via "git log" and looking for the relevant commit message, but you can't just merge 85994 without some extra work.

git lets you create arbitrary refs to arbitrary SHAs, so there is absolutely nothing stopping Boost2Git filling a directory of SHA refs named after each SVN revision e.g. refs/svnrefs/12345 => abcdef...

What is stopping us from doing that is simply that our Boost2Git developers are exhausted. Looking at the "git log" as Jonathan described is just fine for the handful of times this will have to be done. --Beman

Dave Abrahams

21 Oct 21 Oct

3:40 a.m.

on Wed Oct 16 2013, "Niall Douglas" wrote:

...

On 16 Oct 2013 at 17:03, Jonathan Wakely wrote:

...
On 16 October 2013 15:18, Beman Dawes wrote:

...
Like all merges, start by switching to the branch to be merged to - "master" in this case. Then merge from the whole "develop" branch if 85994 is the only difference. Otherwise, you need to merge just the 85994 changeset. (I'm a TortoiseGit user, so am the wrong person to ask about how to do that from the command line. But "git help merge" from the command line may well tell you enough to figure it out.)

I assume 85994 is an svn revision, so to git merge it you need to know the Git SHA1 commit ID for that commit. You can find that via "git log" and looking for the relevant commit message, but you can't just merge 85994 without some extra work.

git lets you create arbitrary refs to arbitrary SHAs, so there is absolutely nothing stopping Boost2Git filling a directory of SHA refs named after each SVN revision e.g. refs/svnrefs/12345 => abcdef...

Neat, but I wonder if we really want to create eighty thousand refs for this purpose.

...

In fact, that is exactly what subgit does: it stores each svn rev to SHA as a svn prop, and each SHA to svn rev as a custom ref. This lets subgit map each to the other transparently along with needed metadata.

Niall

-- Dave Abrahams

Niall Douglas

11 p.m.

On 20 Oct 2013 at 20:40, Dave Abrahams wrote:

...

...
git lets you create arbitrary refs to arbitrary SHAs, so there is absolutely nothing stopping Boost2Git filling a directory of SHA refs named after each SVN revision e.g. refs/svnrefs/12345 => abcdef...

Neat, but I wonder if we really want to create eighty thousand refs for this purpose.

You don't need to include them into the default refs fetch spec i.e. a standard clone won't contain them, but a fetch with the right fetch spec will. In other words, they only consume disc space and bandwidth for those who ask for them. Even then, I'd assume such refs would end up in packed-refs anyway, and so would be highly efficient. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

Dave Abrahams

23 Oct 23 Oct

2:15 a.m.

on Mon Oct 21 2013, "Niall Douglas" wrote:

...

On 20 Oct 2013 at 20:40, Dave Abrahams wrote:

...
...
git lets you create arbitrary refs to arbitrary SHAs, so there is absolutely nothing stopping Boost2Git filling a directory of SHA refs named after each SVN revision e.g. refs/svnrefs/12345 => abcdef...

Neat, but I wonder if we really want to create eighty thousand refs for this purpose.

You don't need to include them into the default refs fetch spec i.e. a standard clone won't contain them, but a fetch with the right fetch spec will.

I know.

...

In other words, they only consume disc space and bandwidth for those who ask for them.

Won't they appear in GitHub's menu and/or network graph?

...

Even then, I'd assume such refs would end up in packed-refs anyway, and so would be highly efficient.

Not that I mind doing this; it'd be a fairly simple change to the tool, but we can always create these afterwards, FWIW, since the revnum is now contained in the commit message. -- Dave Abrahams

Niall Douglas

2:52 a.m.

On 22 Oct 2013 at 19:15, Dave Abrahams wrote:

...

...
In other words, they only consume disc space and bandwidth for those who ask for them.

Won't they appear in GitHub's menu and/or network graph?

You don't have to store refs under refs/heads/ or refs/tags/ which is what GitHub appears to use for display. It would also appear that GitHub only displays a certain depth of nested refs. This can be proved by visiting https://github.com/ned14/boost-trunk/branches where GitHub shows release as a branch, rather than a directory containing many branches (which is surely a bug in GitHub, so I wouldn't rely on this). I would suspect GitHub also stores its own internal refs in magic internal ref locations, so I'd suspect so long as they don't clash you're probably safe. The other option is to set git config to not push the svn refs by default, so they're there for people bothered enough to run Boost2Git for themselves.

...

Not that I mind doing this; it'd be a fairly simple change to the tool, but we can always create these afterwards, FWIW, since the revnum is now contained in the commit message.

Very cool. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

Daniel James

16 Oct 16 Oct

6:22 p.m.

On 16 October 2013 15:18, Beman Dawes wrote:

...

Like all merges, start by switching to the branch to be merged to - "master" in this case. Then merge from the whole "develop" branch if 85994 is the only difference. Otherwise, you need to merge just the 85994 changeset. (I'm a TortoiseGit user, so am the wrong person to ask about how to do that from the command line. But "git help merge" from the command line may well tell you enough to figure it out.)

'git cherry-pick' is the command for merging an individual changeset. These aren't tracked, although git can sometimes work out which commits have been cherry picked (using 'git cherry'). 'git merge' is for merging a branch which is tracked. After conversion, we're probably going to have to spend a little while working out what has been merged for some modules because of the poor merge history (in subversion, not a fault in the conversion). Once that's done merging should hopefully be easier to handle. And FWIW I agree with Beman's plan.

Niall Douglas

4:03 p.m.

On 16 Oct 2013 at 8:51, Beman Dawes wrote:

...

My take on "how quickly can we get out?":

* Declare the conversion software done. Dave and Daniel are off the hook, although we will need their help if the conversion process stops running before the cutover.

* The moment 1.55 ships, we make the SVN repo read only. As soon as the next conversion run finishes, we turn off the conversion process.

* We run a process that checks out all the Boost2Git repos, does the gitflow branch renaming, and pushes those changes up to the Boost2Git repos.

* At that point the bulk conversion is done. Library maintainers can start work on their individual libraries. If a problem is discovered in an individual library's repo, we help that library maintainer resolve the problem for that library rather than rerunning the entire conversion for all libraries.

To be able to pull that off, a lot of preparation not involving the Boost2Git conversion process remains to be done. But we need to be clear about where we are going before worrying about exactly how we are going to get there.

Before any of this, in my opinion we need to find out with real empirical evidence how good the quality of the git conversion is. That, in my opinion, requires a full build and automated test run for every matching git commit and svn commit going back through history to determine their level of equivalence, and at what point in history equivalence breaks down. Without hard data on the git conversion quality, we cannot hope to make informed decisions. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

Dave Abrahams

21 Oct 21 Oct

3:43 a.m.

on Wed Oct 16 2013, "Niall Douglas" wrote:

...

Before any of this, in my opinion we need to find out with real empirical evidence how good the quality of the git conversion is. That, in my opinion, requires a full build and automated test run for every matching git commit and svn commit going back through history to determine their level of equivalence, and at what point in history equivalence breaks down.

Never gonna happen. The filesystem structure is changed, and there's no reason to expect commits in ancient history to build. -- Dave Abrahams

Niall Douglas

11:05 p.m.

On 20 Oct 2013 at 20:43, Dave Abrahams wrote:

...

...
Before any of this, in my opinion we need to find out with real empirical evidence how good the quality of the git conversion is. That, in my opinion, requires a full build and automated test run for every matching git commit and svn commit going back through history to determine their level of equivalence, and at what point in history equivalence breaks down.

Never gonna happen. The filesystem structure is changed, and there's no reason to expect commits in ancient history to build.

I didn't know that was now the case. For reference, I absolutely agree with this decision, but may I ask a stupid question: why now bother converting any history at all if historical checkouts are no longer viable? i.e. why not flatten file history completely, and therefore make clones so very much quicker? Right now my subgit conversion of Boost consumes about 800Mb. Eliminating most of that download would be useful. If people want to see and/or use history from pre-conversion, they can use the SVN repo. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

Paul A. Bristow

22 Oct 22 Oct

8:44 a.m.

...

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Niall Douglas Sent: Tuesday, October 22, 2013 12:05 AM To: boost@lists.boost.org Subject: Re: [boost] [git] [conversion] Schedule and remaining showstoppers?

On 20 Oct 2013 at 20:43, Dave Abrahams wrote:

i.e. why not flatten file history completely, and therefore make clones so very much quicker? Right now my subgit conversion of Boost consumes about 800Mb. Eliminating most of that download would be useful. If people want to see and/or use history from pre-conversion, they can use the SVN repo.

Will this also mean that the very many links in docs to Trac #1234 fixed items will still work without any changes? I worry, ignorantly, that links won't after GIT conversion. Paul --- Paul A. Bristow, Prizet Farmhouse, Kendal LA8 8AB UK +44 1539 561830 07714330204 pbristow@hetp.u-net.com

Dave Abrahams

24 Oct 24 Oct

1:54 a.m.

on Tue Oct 22 2013, "Paul A. Bristow" wrote:

...

...
-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Niall Douglas Sent: Tuesday, October 22, 2013 12:05 AM To: boost@lists.boost.org Subject: Re: [boost] [git] [conversion] Schedule and remaining showstoppers?

On 20 Oct 2013 at 20:43, Dave Abrahams wrote:

i.e. why not flatten file history completely, and therefore make clones so very much quicker? Right now my subgit conversion of Boost consumes about 800Mb. Eliminating most of that download would be useful. If people want to see and/or use history from pre-conversion, they can use the SVN repo.

Will this also mean that the very many links in docs to Trac #1234 fixed items will still work without any changes?

Please show me an example of such a link.

...

I worry, ignorantly, that links won't after GIT conversion.

Paul

--- Paul A. Bristow, Prizet Farmhouse, Kendal LA8 8AB UK +44 1539 561830 07714330204 pbristow@hetp.u-net.com

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

-- Dave Abrahams

Daniel James

8:01 a.m.

On 24 October 2013 02:54, Dave Abrahams wrote:

...

on Tue Oct 22 2013, "Paul A. Bristow" wrote:

...
Will this also mean that the very many links in docs to Trac #1234 fixed items will still work without any changes?

Please show me an example of such a link.

There are lots of such links in the release notes, e.g. in: http://www.boost.org/users/history/version_1_54_0.html Do a text search for #6999, #8421. They will work fine as long as we keep trac running. AFAIK there are no plans to turn it off (but maybe should prevent the creation of new tickets?). But if we ever turn it off, I'm sure we'll set something up to maintain the links. The implicit links in commit logs won't work on github (e.g. trac converts #6999 to a ticket link, r83728 to a link to a changeset, I don't think github offers a way to do that), but since every revision should include a subversion commit number it will be pretty easy to find the changeset on trac where everything will be linked. We could possibly set up our own git viewer to manage that.

Paul A. Bristow

9:09 a.m.

...

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Daniel James Sent: Thursday, October 24, 2013 9:01 AM To: boost@lists.boost.org Subject: Re: [boost] [git] [conversion] Schedule and remaining showstoppers?

On 24 October 2013 02:54, Dave Abrahams wrote:

...
on Tue Oct 22 2013, "Paul A. Bristow" wrote:

...
Will this also mean that the very many links in docs to Trac #1234 fixed items will still work without any changes?

Please show me an example of such a link.

There are lots of such links in the release notes, e.g. in:

http://www.boost.org/users/history/version_1_54_0.html

Do a text search for #6999, #8421.

And there are lots more tickets that *should* also be recorded in their individual docs. Boost.Math is trying to set a good example :-) In Quickbook, the Trac item is linked thus, for example: * Fixed PGI compilation issue [@https://svn.boost.org/trac/boost/ticket/8333 #8333]. And users can quickly see what is work-in-progress thus: Currently open bug reports can be viewed [@https://svn.boost.org/trac/boost/query?status=assigned&status=new&status=reopened&component=math&c ol=id&col=summary&col=status&col=type&col=milestone&col=component&order=priority here].

...

They will work fine as long as we keep trac running. AFAIK there are no plans to turn it off (but maybe should prevent the creation of new tickets?).

So what would replace this very valuable function?

...

But if we ever turn it off, I'm sure we'll set something up to maintain the links.

The implicit links in commit logs won't work on github (e.g. trac converts #6999 to a ticket link,

So these Trac #1234 links will still work as before?

...

r83728 to a link to a changeset, I don't think github offers a way to do that), but since every revision should include a subversion commit number it will be pretty easy to find the changeset on trac where everything will be linked. We could possibly set up our own git viewer to manage that.

I'm reassured by this, but feel that some Trac-like system is very valuable, both for users and developers, so it remains a long-term must-have. Bugs and enhancements are the *first* things that people will be tackling after conversion to GIT (not new libraries at first), so it would be really good if we still had a way of handling the recording of info about the work on them. So (to start with) do we just carry on using Trac as before? Paul --- Paul A. Bristow, Prizet Farmhouse, Kendal LA8 8AB UK +44 1539 561830 07714330204 pbristow@hetp.u-net.com

Daniel James

10:26 a.m.

On 24 October 2013 10:09, Paul A. Bristow wrote:

...

Currently open bug reports can be viewed [@https://svn.boost.org/trac/boost/query?status=assigned&status=new&status=reopened&component=math&c ol=id&col=summary&col=status&col=type&col=milestone&col=component&order=priority here].

You can do something similar with all decent web based bug trackers.

...

...
They will work fine as long as we keep trac running. AFAIK there are no plans to turn it off (but maybe should prevent the creation of new tickets?).

So what would replace this very valuable function?

Possibly github's issue system, if we decide to use it. I was assuming in my last mail we were going to start using it, but I'm not sure if that has actually been decided. IIRC there was some resistance because it's a closed system. Maybe this is a subject for another thread.

...

...
But if we ever turn it off, I'm sure we'll set something up to maintain the links.

The implicit links in commit logs won't work on github (e.g. trac converts #6999 to a ticket link,

So these Trac #1234 links will still work as before?

To be clear, an example is the ticket link in a changeset like: https://svn.boost.org/trac/boost/changeset/86394 That will still be viewable on trac, so the link will work there. You can find the github entry in the history at: https://github.com/boostorg/config/commits/develop As you can see it isn't linked, but has at the end: [SVN r86394] Which can be easily translated to the trac link. We should set up something to make this as easy as possible, either in trac or outside of it.

...

...
r83728 to a link to a changeset, I don't think github offers a way to do that), but since every revision should include a subversion commit number it will be pretty easy to find the changeset on trac where everything will be linked. We could possibly set up our own git viewer to manage that.

I'm reassured by this, but feel that some Trac-like system is very valuable, both for users and developers, so it remains a long-term must-have.

Oh, we'll always use something Trac-like. Whether it's github, continuing with trac, or something else entirely.

...

Bugs and enhancements are the *first* things that people will be tackling after conversion to GIT (not new libraries at first), so it would be really good if we still had a way of handling the recording of info about the work on them.

So (to start with) do we just carry on using Trac as before?

Sure, I was prematurely writing about stopping using Trac in my last mail. I should have been more careful, as I don't know if that's actually the case. Sorry about that.

Philippe Vaucher

11:11 a.m.

...

...
...
They will work fine as long as we keep trac running. AFAIK there are no plans to turn it off (but maybe should prevent the creation of new tickets?).

So what would replace this very valuable function?

Possibly github's issue system, if we decide to use it. I was assuming in my last mail we were going to start using it, but I'm not sure if that has actually been decided. IIRC there was some resistance because it's a closed system. Maybe this is a subject for another thread.

It's probably worth mentionning that we could also import trac tickets/history/file attachments/wiki into redmine, which integrates decently with git repositories and github. I have used different setups in the past, from github being the central repo with a push hook on github that told our redmine server to fetch the last changes, to a redmine-centric workflow where with the help of some plugins users can add their ssh keys to their redmine account and it uses project rights with gitolite, automatic git repository creation on project creation, etc (basically like github, except it's redmine). Not saying this is what boost want, just saying it's an alternative to managing issues on github. Also Trac supports git since a while now, but I think Trac is slowly becoming obsolete. Philippe

Dave Abrahams

25 Oct 25 Oct

3:24 a.m.

On Thursday, October 24, 2013 1:01:17 AM UTC-7, Daniel James wrote:

...

On 24 October 2013 02:54, Dave Abrahams javascript:> wrote:

...
on Tue Oct 22 2013, "Paul A. Bristow"

wrote:

...
...
Will this also mean that the very many links in docs to Trac #1234 fixed items will still work without any changes?

Please show me an example of such a link.

There are lots of such links in the release notes, e.g. in:

http://www.boost.org/users/history/version_1_54_0.html

Do a text search for #6999, #8421.

They will work fine as long as we keep trac running. AFAIK there are no plans to turn it off (but maybe should prevent the creation of new tickets?). But if we ever turn it off, I'm sure we'll set something up to maintain the links.

The implicit links in commit logs won't work on github (e.g. trac converts #6999 to a ticket link,

GitHub does too

...

r83728 to a link to a changeset, I don't think github offers a way to do that),

But not that, as commits are differently-identified. but since every revision

...

should include a subversion commit number it will be pretty easy to find the changeset on trac where everything will be linked. We could possibly set up our own git viewer to manage that.

Possibly.

Daniel James

8:15 a.m.

On 25 October 2013 04:24, Dave Abrahams wrote:

...

On Thursday, October 24, 2013 1:01:17 AM UTC-7, Daniel James wrote:

...
The implicit links in commit logs won't work on github (e.g. trac converts #6999 to a ticket link,

GitHub does too

Not for the trac tickets in historical entries.

Dave Abrahams

2:17 p.m.

Yes, that's understood Sent from my illudium Q-36 explosive space modulator

...

On Oct 25, 2013, at 1:15 AM, Daniel James wrote:

...
On 25 October 2013 04:24, Dave Abrahams wrote:

...
On Thursday, October 24, 2013 1:01:17 AM UTC-7, Daniel James wrote:

The implicit links in commit logs won't work on github (e.g. trac converts #6999 to a ticket link,

GitHub does too

Not for the trac tickets in historical entries.

Daniel James

26 Oct 26 Oct

11:29 a.m.

On 25 October 2013 09:15, Daniel James wrote:

...

On 25 October 2013 04:24, Dave Abrahams wrote:

...
On Thursday, October 24, 2013 1:01:17 AM UTC-7, Daniel James wrote:

...
The implicit links in commit logs won't work on github (e.g. trac converts #6999 to a ticket link,

GitHub does too

Not for the trac tickets in historical entries.

I've tried writing a small chrome extension which creates links for the subversion tags (e.g. [SVN r1234]) in github and bitbucket. It's at: http://www.calamity.org.uk/x/boost-link.crx Because it isn't in the chrome store, it has to be manually installed. To do this download the file, go to the chrome extension page (either through the menu, or at chrome://extensions/), and drag the file into the extension list. It currently only works for the 'boostorg' repositories, as I don't want to mess around with anything unrelated. I might extend it later to handle some of the trac tags. Source code is at: https://github.com/danieljames/boost-link-chrome It turned out to be a little trickier than I thought because of github's javascript navigation. But it seems to work okay.

Dave Abrahams

27 Oct 27 Oct

6:16 p.m.

On Oct 26, 2013, at 4:29 AM, Daniel James wrote:

...

On 25 October 2013 09:15, Daniel James wrote:

...
On 25 October 2013 04:24, Dave Abrahams wrote:

...
On Thursday, October 24, 2013 1:01:17 AM UTC-7, Daniel James wrote:

...
The implicit links in commit logs won't work on Github (e.g. trac converts #6999 to a ticket link,

GitHub does too

Not for the trac tickets in historical entries.

I've tried writing a small chrome extension which creates links for the subversion tags (e.g. [SVN r1234]) in Github and bitbucket. It's at:

http://www.calamity.org.uk/x/boost-link.crx

This is nifty. Maybe someone should turn on the source browser in trac. And maybe you’d also like to rewrite ticket references in commit messages that appear on GitHub, like this one: https://github.com/boostorg/smart_ptr/commit/6eefc6bf81a0edef950e3582a06826f... ?

Dave Abrahams

23 Oct 23 Oct

2:16 a.m.

on Mon Oct 21 2013, "Niall Douglas" wrote:

...

On 20 Oct 2013 at 20:43, Dave Abrahams wrote:

...
...
Before any of this, in my opinion we need to find out with real empirical evidence how good the quality of the git conversion is. That, in my opinion, requires a full build and automated test run for every matching git commit and svn commit going back through history to determine their level of equivalence, and at what point in history equivalence breaks down.

Never gonna happen. The filesystem structure is changed, and there's no reason to expect commits in ancient history to build.

I didn't know that was now the case. For reference, I absolutely agree with this decision, but may I ask a stupid question: why now bother converting any history at all if historical checkouts are no longer viable?

The list demanded it.

...

i.e. why not flatten file history completely, and therefore make clones so very much quicker? Right now my subgit conversion of Boost consumes about 800Mb. Eliminating most of that download would be useful. If people want to see and/or use history from pre-conversion, they can use the SVN repo.

We tried to do that. We even had a solution for people who wanted to graft on an accurate SVN history. It didn't fly. -- Dave Abrahams

Niall Douglas

2:54 a.m.

On 22 Oct 2013 at 19:16, Dave Abrahams wrote:

...

...
...
Never gonna happen. The filesystem structure is changed, and there's no reason to expect commits in ancient history to build.

I didn't know that was now the case. For reference, I absolutely agree with this decision, but may I ask a stupid question: why now bother converting any history at all if historical checkouts are no longer viable?

The list demanded it.

[snip]

We tried to do that. We even had a solution for people who wanted to graft on an accurate SVN history. It didn't fly.

Jeesh. Those people owe you and Daniel a very large and shiny present. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

Robert Ramey

16 Oct 16 Oct

4:42 p.m.

Daniel Pfeifer wrote: It's the hideous complexity of doing a perfect conversion, not

...

...
...
specific issues.

We are not trying to do a perfect conversion. We are trying to do a reasonably good conversion and then move on.

I think we had a reasonably good conversion at BoostCon 2012. Our approach was rejected by the community so we started a new approach, this time aiming for perfect conversion.

The enemy of improvement is the insistence on perfection. Irrelevant question:Will the new system be perfect? Simple question #1: Will we be better off converting than we are now? Simple question #2: Will the improvement be worth the cost in pain an suffering? Robert Ramey

Beman Dawes

5:27 p.m.

On Wed, Oct 16, 2013 at 12:42 PM, Robert Ramey wrote:

...

Daniel Pfeifer wrote: It's the hideous complexity of doing a perfect conversion, not

...
...
...
specific issues.

We are not trying to do a perfect conversion. We are trying to do a reasonably good conversion and then move on.

I think we had a reasonably good conversion at BoostCon 2012. Our approach was rejected by the community so we started a new approach, this time aiming for perfect conversion.

The enemy of improvement is the insistence on perfection.

Irrelevant question:Will the new system be perfect?

Simple question #1: Will we be better off converting than we are now?

Very strong yes to that one!

...

Simple question #2: Will the improvement be worth the cost in pain an suffering?

The pain and suffering is short term. The improvements are long term. --Beman

Dave Abrahams

21 Oct 21 Oct

4:42 a.m.

on Wed Oct 16 2013, Daniel Pfeifer wrote:

...

2013/10/15 Beman Dawes :

...
On Tue, Oct 15, 2013 at 2:41 PM, Niall Douglas wrote:

...
On 15 Oct 2013 at 13:59, Beman Dawes wrote:

...
...
I agree about switching just after a release, but it's not ready for

...
...
...
...
primetime yet IMHO.

Is that a general concern, or are there specific issues you are worried about?

It's the hideous complexity of doing a perfect conversion, not specific issues.

We are not trying to do a perfect conversion. We are trying to do a reasonably good conversion and then move on.

I think we had a reasonably good conversion at BoostCon 2012. Our approach was rejected by the community so we started a new approach, this time aiming for perfect conversion.

The new approach looked very promising at first and we made good progress. But it has become more complex than expected.

You make it sound like it's not working. AFAICT, it's working exactly as we expect it to.

...

It is really like Niall wrote: "Every time you think you're getting closer to done, another new minefield of subtle issues appears which causes more borkage across the whole conversion process."

I disagree strongly. I believe I solved the important problems and the tool is now pretty solid and maintainable. However, I'm not up for making any major changes to it; I just don't have the time. -- Dave Abrahams

Niall Douglas

16 Oct 16 Oct

3:50 p.m.

On 15 Oct 2013 at 17:52, Beman Dawes wrote:

...

...
It's the hideous complexity of doing a perfect conversion, not specific issues.

We are not trying to do a perfect conversion. We are trying to do a reasonably good conversion and then move on.

We want files to appear in the right places, we want to retain history at least for trunk and branches/release, and we want a few other things. But mostly we want to move on and put the conversion behind us. Remember that we will be doing nothing that degrades the svn repo. It will be available for years and years into the future.

Neither Dave nor Daniel appear to think that anything less than a perfect conversion will be acceptable to the Boost community. Everyone here knows how loony I think a perfect conversion is without Dave and Daniel being paid at least the going contracting rate for all those free hours by those who demand a perfect conversion.

...

...
One way I have not previously mentioned here is rebuilding the existing SVN repo to rectify unfortunate historical commits which make perfect git conversion very hard. I know Dave is opposed to this solution, and it certainly launches yet another new tool for converting SVN to cleaner-SVN. But it might be less work overall.

Less work that what? I can't see how adding an additional step to build a new svn repo and then converting that could possibly be less work that finishing off the work that has already been done, and getting on with the actual conversion.

Beman it isn't as easy as that. In git you can't modify any past commit without completely invalidating every single commit thereafter. If after launch someone finds that Boost2Git corrupted some file committed back in 2002, to fix that commit would require rebuilding *every single commit* from 2002 until now. I *absolutely* agree that this is crazy, and in my opinion we should draw a line under how far back the git conversion should go and flatten history before that. Let's say three years: remember, every single commit in the past three years will need checking out and a full build and set of unit tests run, with any differences walked through by human eyes [1]. [1]: Why not just compare the checkouts for equivalence which is much quicker? Because git's checkout and svn's checkout can never be equivalent [2], so Boost2Git employs heuristics to have git checkout something which ought to build and pass unit tests. [2]: Why aren't svn and git checkouts equivalent? Because some files - and we don't know which ones exactly - in svn are incorrectly committed [3], and "just happen" to work in svn but won't in git. Boost2Git fixes some of these up during conversion, hence the loss of bitwise equivalence in the hope of gaining *semantic* equivalence. [3]: This is what I meant by rebuilding the svn repo: if we rebuilt it fixing up all the files which were incorrectly committed in the first place, this solves the mismatch between Boost svn and git.

...

<snip> ... three possible approaches ... </snip>

All I'm seeing is additional complexity and few if any benefits.

It is one thing to fix any remaining glitches, but quite another to re-engineer the whole process because there is fear it might have hidden problems.

I think that's already happened before as Boost2Git is at least the second major attempt at this. And last time I looked (Dave and the Daniel can confirm) the present Boost2Git output has known unknown hidden problems. Until someone does a commit by commit automated bisection to find out how reliable history is (I'm hazarding it will work until the SVN pre-commit filters which exclude bad commits were added), we actually don't know. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

Bjørn Roald

15 Oct 15 Oct

11:38 p.m.

On 10/15/2013 08:41 PM, Niall Douglas wrote:

...

On 15 Oct 2013 at 13:59, Beman Dawes wrote:

...
...
I agree about switching just after a release, but it's not ready for primetime yet IMHO.

Is that a general concern, or are there specific issues you are worried about?

It's the hideous complexity of doing a perfect conversion, not specific issues.

One way I have not previously mentioned here is rebuilding the existing SVN repo to rectify unfortunate historical commits which make perfect git conversion very hard. I know Dave is opposed to this solution, and it certainly launches yet another new tool for converting SVN to cleaner-SVN. But it might be less work overall.

...
...
I would really *strongly* advocate a period where the SVN repo is used for write only, but the git repos are used for everything except writes. Boost2Git has been designed exactly to do that. We should take advantage of it.

How would this work? As a Boost developer how could I use the git repos for my libraries if I can't write to them? How can I use the svn repo if I can't read from it?

I have a fuzzy idea of what you are suggesting, and am interested, but would need to hear specifics of how it could be made to work in practice.

Sure. I'll think aloud here, it may or may not be useful.

Option 1:

I'm thinking that because git is distributed, a Boost developer can merrily commit to their local git without issue. However when one goes to push to origin, a *locally installed* hook script converts the push into a diff patch, and sends that to svn instead.

I'd make it locally installed rather than remotely installed because such a push converting hook script is inevitably brittle, and only those who really know what they're doing should engage.

Disadvantages: Only the brave will bother which reduces number of testers.

Boost2Git changes history in resulting git repositories which invalidate any "local" git repositories you are working in. So as long as the conversion scripts are active, it will be a-lot of noise and complexity if you attempt to get work done in Git. Sorry I don't think this is a very viable approach.

...

Option 2:

Add the metadata to allow subgit (a tool which implements simultaneous svn and git commits) to work with the Boost2Git output. Pushes to git are therefore mirrored by subgit into svn. We'd add a post-push hook to subgit to undo any changes by subgit to the git repo, letting the daily Boost2Git cron job do the mirror instead (unless Boost2Git is upgraded to implement delta updates).

Disadvantages: Would need more work to implement subgit metadata output. Need to buy a subgit licence (they may give one for free in exchange for publicity). Users may be confused if git pushes don't immediately turn up in git origin. git push times are likely to become tens of seconds.

Can subgit deal with history changes caused by Boost2Git?

...

Option 3:

Instead of doing an all-or-nothing SVN to Git conversion, we gradually transition individual libraries out of SVN and into Git submodules using svn:externals i.e. you wipe library X from SVN, replacing it with an svn:externals which loads a Github provided SVN translation from Git (https://help.github.com/articles/support-for-subversion-clients) for that particular library submodule. This lets maintainers test their particular library to make sure it really works as a git submodule, and "detach" its write access from SVN. Slowly, over time, we therefore transition from all-SVN write access, through some-SVN some-Git write access, to all-Git write access, with the SVN repo eventually becoming a "shell" of links to git modules and whose svn revision no longer increments.

Disadvantages: someone has to manage and coordinate this process, and if finding review managers is hard now, finding git conversion managers is probably harder. Explaining it coherently to Boost devs might be hard, especially convincing of need as compared to staying with SVN.

This may be somehow workable approach as it leaves more control to each library maintainer. But I worry about the additional coordination, complexity and management it involves. Why do we need it? I would rather suggest to get all critical processes such as builds, installs and tests verified as working to a reasonable comfort level in git modular "read only" layout while the Boost2Git is active. Then turn Boost2Git off hopefully with reasonable time to next release. Then work out remaining issues in Git. I think this is more or less what Beman suggests. Advantage: This can be done soon, and conversion will be done. There is no plan for how and when all problems with the conversion are discovered and fixed, so waiting for a perfect conversion make no sense. Disadvantage: After general Boost2Git conversion is turned off it is possibly harder to fix history breakage for some library that got it wrong in the last Boost2Git conversion run. Risk mitigation: If seriously broken library history is discovered after general Boost2Git conversion is turned off, that library submodule repository reference can be changed with a simple commit in the boost "master" repository. So it is still possible to fix history for individual libraries as needed with Boost2Git or other tool after the conversion is "completed", but this should only be done if it really is needed. In addition SVN history is available, it does not go away. -- Bjørn

Beman Dawes

16 Oct 16 Oct

5:23 p.m.

On Tue, Oct 15, 2013 at 7:38 PM, Bjørn Roald wrote: On 10/15/2013 08:41 PM, Niall Douglas wrote:

...

Instead of doing an all-or-nothing SVN to Git conversion, we

...
gradually transition individual libraries out of SVN and into Git submodules using svn:externals i.e. you wipe library X from SVN, replacing it with an svn:externals which loads a Github provided SVN translation from Git (https://help.github.com/**articles/support-for-**subversion-clients https://help.github.com/articles/support-for-subversion-clients) for that particular library submodule. This lets maintainers test their particular library to make sure it really works as a git submodule, and "detach" its write access from SVN. Slowly, over time, we therefore transition from all-SVN write access, through some-SVN some-Git write access, to all-Git write access, with the SVN repo eventually becoming a "shell" of links to git modules and whose svn revision no longer increments.

Disadvantages: someone has to manage and coordinate this process, and if finding review managers is hard now, finding git conversion managers is probably harder. Explaining it coherently to Boost devs might be hard, especially convincing of need as compared to staying with SVN.

This may be somehow workable approach as it leaves more control to each library maintainer. But I worry about the additional coordination, complexity and management it involves. Why do we need it?

I would rather suggest to get all critical processes such as builds, installs and tests verified as working to a reasonable comfort level in git modular "read only" layout while the Boost2Git is active. Then turn Boost2Git off hopefully with reasonable time to next release. Then work out remaining issues in Git. I think this is more or less what Beman suggests.

Yes - and spacing out the various actions in my original post to give time for testing makes perfect sense.

...

Advantage: This can be done soon, and conversion will be done. There is no plan for how and when all problems with the conversion are discovered and fixed, so waiting for a perfect conversion make no sense.

Disadvantage: After general Boost2Git conversion is turned off it is possibly harder to fix history breakage for some library that got it wrong in the last Boost2Git conversion run.

Risk mitigation: If seriously broken library history is discovered after general Boost2Git conversion is turned off, that library submodule repository reference can be changed with a simple commit in the boost "master" repository. So it is still possible to fix history for individual libraries as needed with Boost2Git or other tool after the conversion is "completed", but this should only be done if it really is needed. In addition SVN history is available, it does not go away.

Yes, and that pretty much describes my thinking too. --Beman

Daniel Pfeifer

8:56 a.m.

2013/10/15 Niall Douglas :

...

On 15 Oct 2013 at 13:59, Beman Dawes wrote: Option 3:

Instead of doing an all-or-nothing SVN to Git conversion, we gradually transition individual libraries out of SVN and into Git submodules using svn:externals i.e. you wipe library X from SVN, replacing it with an svn:externals which loads a Github provided SVN translation from Git (https://help.github.com/articles/support-for-subversion-clients) for that particular library submodule. This lets maintainers test their particular library to make sure it really works as a git submodule, and "detach" its write access from SVN. Slowly, over time, we therefore transition from all-SVN write access, through some-SVN some-Git write access, to all-Git write access, with the SVN repo eventually becoming a "shell" of links to git modules and whose svn revision no longer increments.

I think that is exactly the approach that was taken by KDE.

Bjørn Roald

12:29 a.m.

On 10/15/2013 02:48 PM, Beman Dawes wrote:

...

It seems to me the best time for the svn to modular git changeover is right after a Boost release. That will give us the maximum time window to get the new house in order before the subsequent release. So should we convert right after 1.55 ships?

I'm well aware that there is a long list of conversion related tasks that we haven't completed. But the svn -> git conversion process itself is so complex that only Dave and Daniel can make modifications. They have done a staggering amount of work already and are nearing exhaustion. Once the conversion is done, there are many of us that understand git well enough to help with problems.

For example, https://github.com/ryppl/Boost2Git/issues/5, "Mimic gitfow", is a showstopper in that what is now "master" should be "develop" and the true "master" (i.e. releases) branch needs to be synthesized. But while it would be great if the Boost2Git process could handle that, it could probably also be do by a post-conversion step performed by git and other command line tools that many of us could help with. Messy, but I'm not willing to take a further delay waiting for perfection.

I think this can be dealt with simply after the conversion is stopped. But it should be among the first things done. To change the Boost2Git conversion tool to make the boost history mimic gitflow does not really make much sense to me. It will sort of work that way anyway as renaming the trunk branch in git to develop is doing exactly that. It get renamed to develop, and work continue there. Alternatively a new branch can be created and the old trunk branch head is left behind. But such historic references such as the last SVN based trunk commit is better dealt with using a tag and delete the old inactive trunk branch head reference.

...

Which of the issues in https://github.com/ryppl/Boost2Git/issues?state=openare true show stoppers?

Are there any true showstoppers missing from that list?

There should be a simple way to find Git commits that originate from specified SVN commits, the simplest way to do this may be to fix https://github.com/ryppl/Boost2Git/issues/38 Record SVN revision numbers somewhere in Git The original svn2git had code to write revision numbers into the commit messages and/or into Git notes. We have neither. As getting the SVN commit IDs into git commits messages will be impossible to do after the final Boost2Git conversion without later manipulating history of all of boost repositories (bad idea), the need for having these IDs in the commit messages need to be addressed now. Other than that I think; https://github.com/ryppl/Boost2Git/issues/30 Deal with branch deletion and maybe; https://github.com/ryppl/Boost2Git/issues/23 Remove redundant merge parents are the only issues in the list that stir some concern with me. But I am not in any way suggesting they are show stoppers. -- Bjørn

Daniel Pfeifer

8:40 a.m.

2013/10/15 Beman Dawes :

...

[Dave and Daniel] have done a staggering amount of work already and are nearing exhaustion.

Actually, we're there.

4197

Age (days ago)

4209

Last active (days ago)

List overview

Download

45 comments

12 participants

participants (12)

Beman Dawes
Bjørn Roald
Daniel James
Daniel Pfeifer
Dave Abrahams
Jonathan Wakely
Niall Douglas
Paul A. Bristow
Peter Dimov
Philippe Vaucher
Robert Ramey
Tom Kent