Provided that the backend repository uses LF line endings exclusively (which is the git-recommended style), autocrlf=true is the recommended setting on Windows and should not result in a diff being shown if the checkout was made with this setting in force. [1][2]

In my experience, it's much simpler to always have autocrlf=false no matter the platform you are on. There are scenarios where autocrlf might be desired, but it's usually much simpler to have it false and handle this at the editor level... http://stackoverflow.com/questions/2825428/why-should-i-use-core-autocrlf-tr... http://stackoverflow.com/questions/2333424/distributing-git-configuration-wi... My 0.02$ Philippe

On Mon, Nov 25, 2013 at 2:40 AM, Gavin Lambert wrote:

On 25/11/2013 14:07, Quoth Dave Abrahams:

Björn Pollex writes:

...

I just cloned the modularized Boost repository from Github [1]. Right

...

after the clone is done, git status gives me a complete diff on bootstrap.bat (so probably line endings).

I do have autocrlf = true in my system configuration, so this shouldn't be the problem, right?

It very well might be the problem.

Provided that the backend repository uses LF line endings exclusively (which is the git-recommended style), autocrlf=true is the recommended setting on Windows and should not result in a diff being shown if the checkout was made with this setting in force. [1][2]

You should only run into trouble if the repository is not LF-only or if your checkout contains mixed-EOL files, either from using editors that altered EOLs (which will be common on Windows if you don't have autocrlf=true) or from changing the autocrlf setting after checking out.

If the conversion is not already normalising line endings to LFs, perhaps it should do so? (With some exceptions for test files specifically for specific EOL handling, if any.)

While setting autocrlf = false fixed the problem for bootstrap.bat, several other files still show changes right after the clone. Here are the steps I executed: git clone https://github.com/boostorg/boost.git git submodule init git submodule update Now when I run git status, I get this output: git status # On branch master # Changes not staged for commit: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # (commit or discard the untracked or modified content in submodules) # # modified: libs/geometry (modified content) # modified: libs/interprocess (modified content) # modified: libs/intrusive (modified content) # modified: libs/multiprecision (modified content) # modified: libs/pool (modified content) # no changes added to commit (use "git add" and/or "git commit -a") The files affected are: libs/geometry/example/ml02_distance_strategy.cpp libs/interprocess/proj/vc7ide/doc_managed_grow.vcproj libs/interprocess/proj/vc7ide/managed_shared_memory.vcproj libs/interprocess/proj/vc7ide/offset_ptr_test.vcproj libs/intrusive/proj/vc7ide/avl_multiset/avl_multiset.vcproj libs/intrusive/proj/vc7ide/avl_set/avl_set.vcproj libs/intrusive/proj/vc7ide/sg_multiset/sg_multiset.vcproj libs/intrusive/proj/vc7ide/sg_set/sg_set.vcproj libs/intrusive/proj/vc7ide/splay_multiset/splay_multiset.vcproj libs/intrusive/proj/vc7ide/splay_set/splay_set.vcproj libs/multiprecision/doc/floating_point_eg1.svg libs/multiprecision/doc/floating_point_eg2.svg libs/multiprecision/doc/floating_point_eg3.svg libs/pool/doc/images/mb1.svg libs/pool/doc/images/mb2.svg libs/pool/doc/images/mb3.svg libs/pool/doc/images/mb4.svg libs/pool/doc/images/pc1.svg libs/pool/doc/images/pc2.svg libs/pool/doc/images/pc3.svg libs/pool/doc/images/pc4.svg libs/pool/doc/images/pc5.svg Except for geometry, all affected files are Visual Studio project files or SVGs. What should be done about this? As far as I can see, these files should not be modified by Git, because these types are explicitly set to "text" in .gitattributes. The same is also true for my previous problem with bootstrap.bat. It is actually strange that changing my autocrlf setting fixed this problem, because .gitattributes should overwrite that setting anyways. Can someone explain this behavior and what I can do to get a clean clone? Regards, Björn Pollex

[1] https://help.github.com/articles/dealing-with-line-endings [2] http://git-scm.com/book/ch7-1.html#Formatting-and-Whitespace

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/ mailman/listinfo.cgi/boost

Gavin Lambert writes:

On 25/11/2013 14:07, Quoth Dave Abrahams:

...

Björn Pollex writes:

...

I just cloned the modularized Boost repository from Github [1]. Right after the clone is done, git status gives me a complete diff on bootstrap.bat (so probably line endings).

I do have autocrlf = true in my system configuration, so this shouldn't be the problem, right?

It very well might be the problem.

Provided that the backend repository uses LF line endings exclusively (which is the git-recommended style), autocrlf=true is the recommended setting on Windows and should not result in a diff being shown if the checkout was made with this setting in force. [1][2]

You should only run into trouble if the repository is not LF-only or if your checkout contains mixed-EOL files, either from using editors that altered EOLs (which will be common on Windows if you don't have autocrlf=true) or from changing the autocrlf setting after checking out.

If the conversion is not already normalising line endings to LFs, perhaps it should do so? (With some exceptions for test files specifically for specific EOL handling, if any.)

[1] https://help.github.com/articles/dealing-with-line-endings [2] http://git-scm.com/book/ch7-1.html#Formatting-and-Whitespace

The conversion represents the file contents from SVN losslessly.

Gavin Lambert writes:

On 27/11/2013 16:54, Quoth Dave Abrahams:

...

Gavin Lambert writes:

...

You should only run into trouble if the repository is not LF-only or if your checkout contains mixed-EOL files, either from using editors that altered EOLs (which will be common on Windows if you don't have autocrlf=true) or from changing the autocrlf setting after checking out.

If the conversion is not already normalising line endings to LFs, perhaps it should do so? (With some exceptions for test files specifically for specific EOL handling, if any.)

[1] https://help.github.com/articles/dealing-with-line-endings [2] http://git-scm.com/book/ch7-1.html#Formatting-and-Whitespace

The conversion represents the file contents from SVN losslessly.

If by that you mean that you are not converting text files to LF-only

We are not.

(or letting git do that for you via eol=lf or via running the conversion on Windows),

Git doesn't do any conversion in the repository. It can do some conversions at checkout time.

then this is probably the problem.

Converting text files to LF-only is also lossless, as long as they really are text files (and you're not relying on the particular EOL format for eg. some test case).

No, it's not lossless. A lossless conversion is reversible. Dropping CRs is clearly not. By the way, some of the files have intentional mixes of CR and LF for testing purposes.

On 27 Nov 2013 at 7:10, Daniel James wrote:

On 27 November 2013 03:54, Dave Abrahams wrote:

...

The conversion represents the file contents from SVN losslessly.

Then the gitattributes file should probably reflect that.

We've already had this discussion on this list several times now. The way conversion is currently represents the lowest amount of data loss. That's why they chose that option. If you don't care too much about corrupting history, I have a lovely patch sitting as a pull request to Boost2Git which will generate you all EOL correct text for all of migrated history. Be aware it does unavoidably eat some data. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

Daniel James writes:

On 27 November 2013 07:18, Niall Douglas wrote:

...

On 27 Nov 2013 at 7:10, Daniel James wrote:

...

On 27 November 2013 03:54, Dave Abrahams wrote:

...

The conversion represents the file contents from SVN losslessly.

Then the gitattributes file should probably reflect that.

We've already had this discussion on this list several times now. The way conversion is currently represents the lowest amount of data loss. That's why they chose that option.

I'm not saying the conversion should be changed. I'm saying that the gitattributes file should represent the conversion. It currently doesn't which is why there are problems. It's claiming that "*.bat", "*.svg" and "*.vsproj" files have normalised line endings, but they don't and that confuses git.

The .gitattributes file is part of the conversion. Its exact contents can be changed at the cost of another conversion run.

AMDG On 11/25/2013 09:28 AM, Beman Dawes wrote:

I also had autocrlf = true on my Windows machine, so set it to false and tried a fresh clone.

.\bootstrap failed! Couldn't find compiler configuration files. Oddly, bootstrap.bat still had cr/lf line endings, and was identical to the version cloned with autocrlf = true, even though many other files differed in line endings.

What about tools/build/v2/engine/build.bat (Which does most of the real work)? In Christ, Steven Watanabe

AMDG On 11/25/2013 10:24 AM, Beman Dawes wrote:

Ha! tools/build/v2/engine/build.bat just has newlines. Yet it has cr/lf line endings in svn.

So that's why on Windows autocrlf = true is required. It also shows that conversion's treatment of line endings is inconsistent - apparently bootstrap.bat does not have CR stripped but tools/build does have them stripped.

The difference appears to because:

svn propget svn:eol-style trunk\bootstrap.bat CRLF

svn propget svn:eol-style trunk\tools\build\v2\engine\build.bat native

It sounds like the conversion doesn't handle svn:eol-style=CRLF correctly. In Christ, Steven Watanabe

AMDG On 11/27/2013 08:03 AM, Dave Abrahams wrote:

What do you consider to be "correct" handling?

If you can figure out what's really desired, we can easily implement it. Or, maybe Niall's patch already implements what you want. So far, though, there have been so many emphatic demands from the community that no history be "lost," that we have been extremely reluctant to process the file contents as part of the conversion.

That is exactly the problem. You /are/ losing part of the history. There are really two options: a) Correctly preserve svn:eol-style, however git represents it, so that the files produced by a git checkout are always binary identical to those produced by svn checkout. b) Change the line endings to match whatever the rules are in the git repository. I would strongly favor option (a), unless git is too broken to handle it. In Christ, Steven Watanabe

AMDG On 11/27/2013 09:31 AM, Dave Abrahams wrote:

...

There are really two options: a) Correctly preserve svn:eol-style, however git represents it,

Git doesn't represent it.

...

so that the files produced by a git checkout are always binary identical to those produced by svn checkout.

How things are checked out by Git depends on a user's local settings (e.g. autocrlf), so there is no way we can represent things so that a git checkout is *always* identical to an SVN checkout.

(Is the same true of SVN?)

Do I really have to spell this out explicitly? I thought it would be obvious that (assuming I understand how this works in git) autocrlf=true, would mean that the git checkout should be identical to an svn checkout on a platform where the native line ending is "\r\n" and aurocrlf=false would mean that it corresponds to a platform with "\n" line endings. I don't really care whether this is decided via a configuration option or by platform. Any other options that affect file contents and have associated metadata in svn, should be handled similarly. Anyway, this property needs to be preserved (I believe that what I just described is actually satisfied), while /also/ satisfying the rule: - A fresh git checkout shall not show any uncommitted changes. In Christ, Steven Watanabe

On 11/27/2013 2:01 PM, Dave Abrahams wrote:

Steven Watanabe writes:

...

AMDG

On 11/27/2013 09:31 AM, Dave Abrahams wrote:

...

...

There are really two options: a) Correctly preserve svn:eol-style, however git represents it,

Git doesn't represent it.

...

so that the files produced by a git checkout are always binary identical to those produced by svn checkout.

How things are checked out by Git depends on a user's local settings (e.g. autocrlf), so there is no way we can represent things so that a git checkout is *always* identical to an SVN checkout.

(Is the same true of SVN?)

Do I really have to spell this out explicitly?

Could you make an effort to use a slightly less contemptuous tone? Obviously I am not smart enough to deduce what you mean without an explicit spelling-out, or I wouldn't be asking.

...

I thought it would be obvious that (assuming I understand how this works in git) autocrlf=true, would mean that the git checkout should be identical to an svn checkout on a platform where the native line ending is "\r\n" and aurocrlf=false would mean that it corresponds to a platform with "\n" line endings.

So your criteria for success are:

* a user with autocrlf=true set, on any platform, sees the same file contents as an SVN user would on Windows, and * a user with autocrlf=false set sees the same file contents as an SVN user would on Linux

?

I'm not even sure that's achievable. It's my understanding that autocrlf=true is supposed to "do the right thing" depending on platform, so a Unix user sees unix line endings and a Windows user sees windows line endings.

...

I don't really care whether this is decided via a configuration option or by platform. Any other options that affect file contents and have associated metadata in svn, should be handled similarly.

I'm sorry, but for me to make changes to the process, I need to hear things spelled out more precisely than "handled similarly," if for no other reason than that I need to make sure the rest of the people on the list understand things the same way you and I do. I would hate to satisfy some mutual understanding we develop and then find that the rest of the community disagrees with that choice.

...

Anyway, this property needs to be preserved (I believe that what I just described is actually satisfied), while /also/ satisfying the rule: - A fresh git checkout shall not show any uncommitted changes.

No matter what your personal settings?

I do not know git very well. I think the basic issue is that in SVN when one checks out a text file from the repository, line endings are converted from whatever they are in the repository to whatever the OS normally uses; for Windows CR-LF, for Linux LF. SVN does not think that the file has been changed because of this. Then when one commits a file in SVN the line ending is changed back to whatever it was originally in the SVN repository when it is put back in the repository. One would like git to work the same way. I have no idea whether it does or not. But it seems irrelevant from my point of view what the line endings are in a repository as long as when I work on a file locally from a git repostory the line endings are correct for the OS I am on and git does not think there is some change just because the line endings in the repository and the line endings in the file locally are different. I hope git can accomplish this in some way, whether through some local settings or whatever.

On 28 November 2013 11:35, Philippe Vaucher wrote:

Do people really want this behavior? It looks so hacky/kludgy to me. I mean, nowadays all editors can happily edit files with LF or CRLF and save them that way too. What do you gain from that behavior, except extra configuration of the tools and more headaches? I understand from where this feature comes from, back in the days where editors couldn't handle different line endings, but this is not the case anymore.

Nowadays line ending problems aren't with text editors. Some unix tools require LF for end of line, and on windows batch files require CRLF. And they often don't fail gracefully. The files where this is a problem are mainly platform specific files, although also SVG files because subversion treats them as binary files. We can get this right by setting the gitattributes files to something appropriate and normalizing the repos (most repos are actually fine already). But there will be issues with checking out old versions. There are a few approaches to fixing this, but Beman isn't keen on delaying conversion further. Most people don't use old versions so it won't be a big problem, and if they wish to they can use the read only subversion repo, or normalise an older version themselves.

On 28 Nov 2013 at 10:14, Beman Dawes wrote:

...

We can get this right by setting the gitattributes files to something appropriate and normalizing the repos (most repos are actually fine already). But there will be issues with checking out old versions. There are a few approaches to fixing this, but Beman isn't keen on delaying conversion further. Most people don't use old versions so it won't be a big problem, and if they wish to they can use the read only subversion repo, or normalise an older version themselves.

+1

In which case, we merge https://github.com/ryppl/Boost2Git/pull/42 and run one last conversion. All commits made to Boost since the SVN pre-commit hooks were added will be converted losslessly (i.e. last few years worth of commits). All commits preceding this where the EOLs do not match the file extension in the .gitattributes will be forced to the EOL specified in .gitattributes. For those test files in sandbox where deliberate mismatched EOLs are part of a test suite, those files will be irretrievably forced to correct EOL and therefore the unit tests will fail. Other similar loss of data may occur in files committed a long time ago. This is therefore the choice for the Boost community: (i) Correct EOLs in all commits made last few years AND accept some data loss in prior (mostly ancient) commits OR (ii) mismatched EOLs during Boost checkout from git on Windows. OR (iii) Someone in the Boost community comes up with something better which can be implemented before the permanent switch over next week. Your choice, as Dave pointed out. We need consensus from this list on what to do. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

On 29 November 2013 04:18, Niall Douglas wrote:

On 28 Nov 2013 at 10:14, Beman Dawes wrote:

...

...

We can get this right by setting the gitattributes files to something appropriate and normalizing the repos (most repos are actually fine already). But there will be issues with checking out old versions. There are a few approaches to fixing this, but Beman isn't keen on delaying conversion further. Most people don't use old versions so it won't be a big problem, and if they wish to they can use the read only subversion repo, or normalise an older version themselves.

+1

In which case, we merge https://github.com/ryppl/Boost2Git/pull/42 and run one last conversion. All commits made to Boost since the SVN pre-commit hooks were added will be converted losslessly (i.e. last few years worth of commits). All commits preceding this where the EOLs do not match the file extension in the .gitattributes will be forced to the EOL specified in .gitattributes. For those test files in sandbox where deliberate mismatched EOLs are part of a test suite, those files will be irretrievably forced to correct EOL and therefore the unit tests will fail. Other similar loss of data may occur in files committed a long time ago.

When I said, "There are a few approaches to fixing this, but Beman isn't keen on delaying conversion further." I was including this approach.

On Thu, Nov 28, 2013 at 3:35 PM, Philippe Vaucher wrote:

...

I think the basic issue is that in SVN when one checks out a text file from the repository, line endings are converted from whatever they are in the repository to whatever the OS normally uses; for Windows CR-LF, for Linux LF. SVN does not think that the file has been changed because of this. Then when one commits a file in SVN the line ending is changed back to whatever it was originally in the SVN repository when it is put back in the repository.

One would like git to work the same way. I have no idea whether it does or not. But it seems irrelevant from my point of view what the line endings are in a repository as long as when I work on a file locally from a git repostory the line endings are correct for the OS I am on and git does not think there is some change just because the line endings in the repository and the line endings in the file locally are different. I hope git can accomplish this in some way, whether through some local settings or whatever.

Do people really want this behavior? It looks so hacky/kludgy to me. I mean, nowadays all editors can happily edit files with LF or CRLF and save them that way too. What do you gain from that behavior, except extra configuration of the tools and more headaches? I understand from where this feature comes from, back in the days where editors couldn't handle different line endings, but this is not the case anymore.

New files are commonly created with native line endings. I'm not very knowledgeable about git, but if it stores files as-is without any conversions, then we may have a situation that someone adds and commits a file with CRLF on Windows and someone else checks it out on Linux and sees the incorrect line endings. It may be less harmful if it happens the other way around, but I'd say you shouldn't rely on that everyone on Windows uses smart enough editors that don't convert line ending on saving the file. There has to be some protection against these issues, either in the form of svn-like conversion or some hooks that don't allow pushing files with incorrect line endings to github. I don't know if the latter is possible though.

On 11/28/2013 6:35 AM, Philippe Vaucher wrote:

...

I think the basic issue is that in SVN when one checks out a text file from the repository, line endings are converted from whatever they are in the repository to whatever the OS normally uses; for Windows CR-LF, for Linux LF. SVN does not think that the file has been changed because of this. Then when one commits a file in SVN the line ending is changed back to whatever it was originally in the SVN repository when it is put back in the repository.

One would like git to work the same way. I have no idea whether it does or not. But it seems irrelevant from my point of view what the line endings are in a repository as long as when I work on a file locally from a git repostory the line endings are correct for the OS I am on and git does not think there is some change just because the line endings in the repository and the line endings in the file locally are different. I hope git can accomplish this in some way, whether through some local settings or whatever.

Do people really want this behavior? It looks so hacky/kludgy to me. I mean, nowadays all editors can happily edit files with LF or CRLF and save them that way too. What do you gain from that behavior, except extra configuration of the tools and more headaches? I understand from where this feature comes from, back in the days where editors couldn't handle different line endings, but this is not the case anymore.

Please stop acting as if this is not an issue. Your naivete about how tools, whether on Windows, Linux, or Macintosh work with any type of line endings is just wrong. Sure it would be nice if this was not an issue, but that is not reality.

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Paul Mensonides Sent: Thursday, November 28, 2013 9:17 PM To: boost@lists.boost.org Subject: Re: [boost] [git] Line endings in bootstrap.bat on Windows

On 11/28/2013 9:48 AM, Edward Diener wrote:

...

Please stop acting as if this is not an issue. Your naivete about how tools, whether on Windows, Linux, or Macintosh work with any type of line endings is just wrong. Sure it would be nice if this was not an issue, but that is not reality.

As much as I dislike Windows' CRLF, +1 to the above. It isn't just a matter of editors, but all text- processing tooling. Not even close to "doesn't matter anymore".

+1 It's got to 'just work' on Windows, Linux, or Macintosh. Paul PS (And I fear there are going to be lots more things to fix to get the (re-)generation of documentation working?) --- Paul A. Bristow, Prizet Farmhouse, Kendal LA8 8AB UK +44 1539 561830 07714330204 pbristow@hetp.u-net.com

...

PS (And I fear there are going to be lots more things to fix to get the (re-)generation of documentation working?)

I'm going to try to build the documentation from git this weekend. I'll write about it on the documentation list.

One of the issues is that there are lots of links from docs to headers - probably a search and replace to fix, but some thought might have to go into what the right thing to do actually is...? John.

On 11/28/2013 10:17 PM, Paul Mensonides wrote:

On 11/28/2013 9:48 AM, Edward Diener wrote:

...

Please stop acting as if this is not an issue. Your naivete about how tools, whether on Windows, Linux, or Macintosh work with any type of line endings is just wrong. Sure it would be nice if this was not an issue, but that is not reality.

As much as I dislike Windows' CRLF, +1 to the above. It isn't just a matter of editors, but all text-processing tooling. Not even close to "doesn't matter anymore".

I don't think anybody imply that this shall not be fixed, at the very latest as soon as the final conversion is done and Git repositories are live. The question is whether it is worth the risk, time, and effort involved in running the conversion again with these changes applied to historic commits in the git repositories. It is unlikely many people, if any, will build passed versions of boost from modularized repositories. If they do they can apply the needed changes for CRLF along with a number of other tweaks that are needed to make things work. my $0.02 -- Bjørn

Beman Dawes writes:

On Mon, Nov 25, 2013 at 12:40 PM, Steven Watanabe wrote:

...

AMDG

On 11/25/2013 09:28 AM, Beman Dawes wrote:

...

I also had autocrlf = true on my Windows machine, so set it to false and tried a fresh clone.

.\bootstrap failed! Couldn't find compiler configuration files. Oddly, bootstrap.bat still had cr/lf line endings, and was identical to the version cloned with autocrlf = true, even though many other files

differed

...

in line endings.

What about tools/build/v2/engine/build.bat (Which does most of the real work)?

Ha! tools/build/v2/engine/build.bat just has newlines. Yet it has cr/lf line endings in svn.

So that's why on Windows autocrlf = true is required. It also shows that conversion's treatment of line endings is inconsistent

That is incorrect. There is *no* CRLF handling in the conversion process right now. It converts bytes literally.

- apparently bootstrap.bat does not have CR stripped but tools/build does have them stripped.