Hi Gavin, On Tuesday, 26. November 2013 20:06:00 Gavin Lambert wrote:

On 26/11/2013 19:54, Quoth Jürgen Hunold:

...

boost.svn$ svn proplist libs/pool/doc/images/mb1.svg

Properties on 'libs/pool/doc/images/mb1.svg': svn:mime-type

boost.svn$ svn propget svn:mime-type libs/pool/doc/images/mb1.svg image/svg+xml

And image/* marks this as "binary". We had this discussion weeks ago with noch real decision.

Just curious: where is that rule coming from? If it's in the conversion code (which seems likely), can it be changed to treat */*+xml as text regardless of other rules?

No, subversion will not do CRLF-conversion on binary files. This creates the same issues as explicit CRLF settings when doing the conversion on Unix. And the conversion script will have to use subversion commands in order to get the commits. The reason for the "image/*" setting for svg was to have them displayed as images when viewed in a web browser. But those display settings can be better configured server-side. My experience is to have all text file eol-style "native" and mime-type "text/something" to get the best cross-platform integration. This is true even for .vcproj files as you can then script-edit them on Unix without problems.

...

The final question is how to resolve this. I am not sure which solution is best. I'd like to set all text-based files (including .svg and .bat) to svn:eol-style native and svn:mime-type text/* in svn and then re-run the conversion again. The main reason is that this worked really well for the rest of the repository.

That might be easier though, provided they're all found. :)

We have to handle all files which come out "modified" from a fresh checkout. The list from Beman is quite comprehensive. And I would just use a skript to force-set the needed properties on the svn side. Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! * voice: ++49 4257 300 ! Fährstraße 1 * fax : ++49 4257 300 ! 31609 Balge/Sebbenhausen * jhunold@gmx.eu ! Germany

On 26/11/2013 20:18, Quoth Jürgen Hunold:

...

The reason for the "image/*" setting for svg was to have them displayed as images when viewed in a web browser. But those display settings can be better configured server-side. My experience is to have all text file eol-style "native" and mime-type "text/something" to get the best cross-platform integration. This is true even for .vcproj files as you can then script-edit them on Unix without problems.

Changing subversion properties doesn't fix historical data, so it won't fix anything that we can't fix in git. On 26 November 2013 23:34, Gavin Lambert wrote:

Another possibility might be to explicitly mark all text files in .gitattributes with eol=lf and then run the conversion; I think that would fix the repository blobs, and then the eol attribute can be removed for actual use.

The conversion doesn't respect .gitattributes, that's why there's a problem. We can fix these issues in current and future versions by normalizing the files (as described in the gitattributes man page). Since we can't fix these files in historical versions we could re-run the conversion with gitattributes that unsets the text attribute for the problematic files (something like "*.bat -text", "*.vsproj -text", "*.svg -text"). Then in git update the gitattributes (to something like "*.bat text eol=crlf", "*.vsproj text eol=crlf", "*.svg text") and normalize the files so that it does what we want. We can script that, so it shouldn't require too much work.

On Wed, Nov 27, 2013 at 4:43 AM, Philippe Vaucher < philippe.vaucher@gmail.com> wrote:

Excuse my ignorance, but why do we need to set attributes about eol or even suggest auto.crlf=true ? Why can't this be managed at the editor level?

The core.autocrlf true suggestion for Windows users is a bit of a side track, and will probably be removed. The .attributes file allows "you to ensure consistent behaviour for all users regardless of their git settings." See https://help.github.com/articles/dealing-with-line-endings#platform-all At least that's my current understanding, so take it with a grain of salt. --Beman

Hi Beman, On Wednesday, 27. November 2013 09:25:19 Beman Dawes wrote:

On Wed, Nov 27, 2013 at 2:00 AM, Daniel James wrote:

...

Changing subversion properties doesn't fix historical data, so it won't fix anything that we can't fix in git.

That's also my understanding.

Yes, this is right.

...

The conversion doesn't respect .gitattributes, that's why there's a problem. We can fix these issues in current and future versions by normalizing the files (as described in the gitattributes man page).

Ah, "normalizing" is the keyword I am missing. Thanks.

After doing a lot of reading of git docs and about how others dealt with similar problems, that was also my conclusion. I'm wondering why others haven't suggested renormalization?

I'm still more familiar with Subversion than git. Fixing it on the svn side would be my personal favourite. I did some similar conversions on while converting from CVS years ago in order to end up with a "clean" repository without need for further actions. But cvs2svn/git is a scary thing...

Since we can't fix these files in historical versions we could re-run

...

the conversion with gitattributes that unsets the text attribute for the problematic files (something like "*.bat -text", "*.vsproj -text", "*.svg -text"). Then in git update the gitattributes (to something like "*.bat text eol=crlf", "*.vsproj text eol=crlf", "*.svg text") and normalize the files so that it does what we want. We can script that, so it shouldn't require too much work.

As long as we are sure the .gitattributes are correct, and then renormalize, why do we have to rerun the conversion at all? What am I missing?

My goal was to avoid any "conversion" commits. I should have explicitly said so. I'm fine with having a handful of "normalizing" commits and let the repository go live.

From the replies I think there is consensus (or at least a great majority) to renormalize and move forward.

Thanks for managing this whole thing so patiently. Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! * voice: ++49 4257 300 ! Fährstraße 1 * fax : ++49 4257 300 ! 31609 Balge/Sebbenhausen * jhunold@gmx.eu ! Germany

On Wed, Nov 27, 2013 at 10:47 AM, Daniel James wrote:

On 27 November 2013 14:25, Beman Dawes wrote:

...

...

Since we can't fix these files in historical versions we could re-run the conversion with gitattributes that unsets the text attribute for the problematic files (something like "*.bat -text", "*.vsproj -text", "*.svg -text"). Then in git update the gitattributes (to something like "*.bat text eol=crlf", "*.vsproj text eol=crlf", "*.svg text") and normalize the files so that it does what we want. We can script that, so it shouldn't require too much work.

As long as we are sure the .gitattributes are correct, and then renormalize, why do we have to rerun the conversion at all? What am I missing?

If someone checks out an older version of a module (i.e. before normalization) then they'll still have this problem. If the conversion is rerun with gitattributes set as I described then I don't think they will. It won't fix the files, it'll just tell git not to expect them to have the correct newlines.

Ah! Understood. But like you, I don't see that as a big deal.

But this isn't a big deal, and there are possibly issues with this approach, so it might not be worth it.

I would really like to avoid rerunning the conversion, since just to be sure I would want to then rerun the independent verification checks. That involves a lot of semi-manual steps, and it has taken me the better part of three days so far. I'm having to rerun the trunk/develop verification right now because it ran out of disk space last night. Sigh. I'd also like to hear other opinions, particularly from Dave, on this before we make a final decision. --Beman

On Wed, Nov 27, 2013 at 11:25 AM, Dave Abrahams wrote:

Beman Dawes writes:

...

After doing a lot of reading of git docs and about how others dealt with similar problems, that was also my conclusion. I'm wondering why others haven't suggested renormalization?

They have.

...

...

Since we can't fix these files in historical versions we could re-run the conversion with gitattributes that unsets the text attribute for the problematic files (something like "*.bat -text", "*.vsproj -text", "*.svg -text"). Then in git update the gitattributes (to something like "*.bat text eol=crlf", "*.vsproj text eol=crlf", "*.svg text") and normalize the files so that it does what we want. We can script that, so it shouldn't require too much work.

As long as we are sure the .gitattributes are correct, and then renormalize, why do we have to rerun the conversion at all? What am I missing?

It depends on whether you want to "correct" history or not. If you want to go back in time and renormalize, then all the SHAs in submodules change, and the branches in the master repo need to be rewritten accordingly.

My initial reaction is not to "correct" history. I'm quite comfortable with the idea that files from svn were moved into "git" as is, and then as part of bringing Boost into the git world, we renormalized. I think of it this way; a programmer deeply familiar with git who has never seen Boost before should not be surprised necessarily, beyond having to grasp the idea that Boost is a set of individual libraries rather than on giant library. IIUC, such a programmer would expect normalized line endings in master and develop heads, and .attributes files to ensure portability. --Beman

On 29 November 2013 04:04, Niall Douglas wrote:

On 27 Nov 2013 at 9:25, Beman Dawes wrote:

...

After doing a lot of reading of git docs and about how others dealt with similar problems, that was also my conclusion. I'm wondering why others haven't suggested renormalization?

This was discussed and rejected due to loss of data. Here is the patch implementing renormalisation: https://github.com/ryppl/Boost2Git/pull/42.

When we talk about renormalisation, we're talking about the procedure described in the gitattributes man page.

On 29 Nov 2013 at 8:14, Beman Dawes wrote:

...

When we talk about renormalisation, we're talking about the procedure described in the gitattributes man page.

Unfortunately that procedure would corrupt many files within Boost.

+1

There is also a nice discussion at https://help.github.com/articles/dealing-with-line-endings#re-normalizing-a-...

I've forked the boost super repo and am testing the procedure now.

Off the top of my head, you'll need to watch for the following (this list is incomplete): * Files with text file extensions not in ASCII or UTF-8. If you use simple EOL renormalisation with UTF-16 text for example, you'll corrupt that text. * Text files with intentionally mixed EOLs. You'll need to change their extension to not .txt (best), or add special exceptions to .gitattributes (brittle, I wouldn't recommend this option). * Scan the first 8Kb of every file with an extension not marked as text nor binary in .gitattributes for zeros. If you don't find a zero, git will assume it is text and EOL normalise it. Unfortunately some binary file types such as PDF don't have zeros in their first 8Kb, so that would be very bad. We never dealt with these issues during conversion, and there are probably more we don't know about yet. This is why I said Boost is not ready to do the transition - plus too few want to do the manual labour involved in achieving a "perfect" conversion and just want "someone else" to do the tedious work for them. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

On 29 Nov 2013 at 16:26, Beman Dawes wrote:

...

...

...

When we talk about renormalisation, we're talking about the procedure described in the gitattributes man page.

Unfortunately that procedure would corrupt many files within Boost.

I've tested, on Windows and Linux, without apparent problems. The most files modified (152) are on master. Here is the list:

I looked through that list - it doesn't seem to contain anything from Sandbox? My original Boost2Git EOL conversion patch was developed against trunk only i.e. not including sandbox. That conversion, to my knowledge, was completely lossless, even when going right back to the beginning. The problem was Sandbox really. Dave spotted that my patch fatal exited when Sandbox was included, and it was an assertion check for an "impossible" condition. Things got worse the more I worked around problems. I eventually concluded we would have to accept data loss if we were converting all of history, and upcalled the decision to Dave and Daniel. One option I did posit was to convert the last three years of history, and flatten everything before that. We could have done a perfect conversion of the last three years easily enough, and it was my personally preferred option. I think we're beyond that now though.

...

Off the top of my head, you'll need to watch for the following (this list is incomplete): [snip] None of the modified files fit in that category.

I checked tools/inspect/wrong_line_ends_test.cpp to see why it wasn't normalized, and the reason was simple. It had already been normalized! That isn't worrisome - inspect is a tool that will need tuning for git anyhow.

The problematic files were mostly in Sandbox. I found little issue with trunk. I think because stuff in trunk had to pass peer review, people didn't do unwise commits.

.pdf is in .gitattributes, so no problem. I checked a couple of .pdf files to be sure, and adobe reader opens them without problems.

Yes, I made sure .pdf was in gitattributes specifically to deal with that problem. I still think a test which sweeps the first 8Kb of all files in Boost which don't have extensions in .gitattributes is a very good idea.

...

We never dealt with these issues during conversion, and there are probably more we don't know about yet. This is why I said Boost is not ready to do the transition - plus too few want to do the manual labour involved in achieving a "perfect" conversion and just want "someone else" to do the tedious work for them.

I'm sure there are problems we don't know about. But we have done enough testing do know that vast numbers of files were converted correctly, that passing tests on both trunk and branches/release still pass, and that the small number of the minor problems we have found are not even close to being showstoppers.

To delay further just because of FUD will be harmful.

Caution is not FUD. My biggest single worry has always been lack of testing of the validity of the conversion. I personally hate doing anything irreversible which has not been tested to destruction. The decision is out of my hands of course. And I have always seen your point about getting on with it Beman, it's just I am personally much more cautious in this type of situation (equally I am far less cautious than many on this list in many other situations). Also, it's entirely possible much more testing has been done than I am aware of - after all I am not on the steering commitee. If that is so, my caution is unwarranted and I have been talking out of my ass.

Thanks for your list of possible problem areas. It gave me something additional things to look for.

You're welcome. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

Gavin Lambert writes:

I'm just suggesting that (if it makes sense with how the conversion tool actually works -- I don't know much about it) a higher priority rule be added to the conversion tool that says "mime type */*+xml in svn => text attribute in git", which would then result in SVG files being marked as text and all other files marked as images.

Such changes can be made.

Though it'd still require the files to be dos2unix'd if the conversion was run on Linux

As mentioned elsewhere, that's incorrect. The files are never checked out into a working copy during the conversion process. That would be hella-slow.

-- if it was run on Windows then the text attribute by itself should fix the problem.

Another possibility might be to explicitly mark all text files in .gitattributes with eol=lf and then run the conversion; I think that would fix the repository blobs,

What goes into .gitattributes has no effect on the content of blobs in the repository.

and then the eol attribute can be removed for actual use.

Jürgen Hunold writes:

Hi Gavin,

On Tuesday, 26. November 2013 20:06:00 Gavin Lambert wrote:

...

On 26/11/2013 19:54, Quoth Jürgen Hunold:

...

boost.svn$ svn proplist libs/pool/doc/images/mb1.svg

Properties on 'libs/pool/doc/images/mb1.svg': svn:mime-type

boost.svn$ svn propget svn:mime-type libs/pool/doc/images/mb1.svg image/svg+xml

And image/* marks this as "binary". We had this discussion weeks ago with noch real decision.

Just curious: where is that rule coming from? If it's in the conversion code (which seems likely), can it be changed to treat */*+xml as text regardless of other rules?

No, subversion will not do CRLF-conversion on binary files. This creates the same issues as explicit CRLF settings when doing the conversion on Unix.

Let me be very clear: it does not matter a bit what platform is used to do the conversion. The bytes of every file are streamed literally from SVN to Git. https://github.com/ryppl/Boost2Git/blob/master/src/importer.cpp#L459

And the conversion script

And please don't call it a script ;-)

will have to use subversion commands in order to get the commits.

The conversion works uses the SVN API directly.

The reason for the "image/*" setting for svg was to have them displayed as images when viewed in a web browser. But those display settings can be better configured server-side. My experience is to have all text file eol-style "native" and mime-type "text/something" to get the best cross-platform integration. This is true even for .vcproj files as you can then script-edit them on Unix without problems.

...

...

The final question is how to resolve this. I am not sure which solution is best. I'd like to set all text-based files (including .svg and .bat) to svn:eol-style native and svn:mime-type text/* in svn and then re-run the conversion again. The main reason is that this worked really well for the rest of the repository.

That might be easier though, provided they're all found. :)

We have to handle all files which come out "modified" from a fresh checkout. The list from Beman is quite comprehensive. And I would just use a skript to force-set the needed properties on the svn side.

Yours,

Jürgen