The right way to handle this is to submit a pull request against https://github.com/ryppl/Boost2Git/blob/master/dot_gitattributes Sent from my moss-covered three-handled family gradunza On Jul 31, 2013, at 2:14 AM, Jürgen Hunold wrote:

Hi Vitaly,

I'm CC#ing this to Dave as this might go below the radar.

On Tuesday, 30. July 2013 00:33:53 Vitaly Budovski wrote:

...

Please see the attached files. boost_endings1.patch just sets some of the cpp and other text files to have file endings consistent with the rest of the repository.

Patch applies, but I think svn:eol-styles might be missing.

...

boost_endings2.patch may be controversial as it does the same but for the svg files. Some of them are marked as text, others as binary. It's a real mess. Most of the svg files are generated from other sources so should probably be omitted from the repository and simply regenerated when docs are built.

eol-style setting here again. Please note that generated documentation is allowed to be checked in.

...

Came across these issues when checking out the modularised boost git repository.

Right, please add [git] to your subject next time to get attention of the conversion team.

For the record: Those line endings lead to instant modified git checkout when using modularised boost. I think this is due to the conversion settings and/or missing svn:eol-style.

Yours,

Jürgen

-- * Dipl.-Math. Jürgen Hunold ! * voice: ++49 4257 300 ! Fährstraße 1 * fax : ++49 4257 300 ! 31609 Balge/Sebbenhausen * jhunold@gmx.eu ! Germany

on Wed Jul 31 2013, Vitaly Budovski wrote:

Hi Juergen,

I've had a look and the svn:eol-style property seems to be present in the patches. Perhaps I've missed a file or two?

Dave,

I've had a look at the dot_gitattributes file and there doesn't seem to be anything wrong with it. The main issue is that there are a lot of inconsistencies with files of the same extension, so pretty much no matter what you change in the .gitattributes you will end up with issues. For the svg, we could change the type to binary, but that would potentially create a lot of noisy commits if people regenerate them on different platforms. It seems like the only proper fix is to correct the problems at the source.

Because the git repositories are continuously rebuilt from SVN, correcting the problems at the source means checking fixes into SVN. I support that approach. You don't need anything from me in order to do that; just do it! -- Dave Abrahams

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Jürgen Hunold Sent: Wednesday, August 07, 2013 10:55 AM To: boost@lists.boost.org Cc: Dave Abrahams Subject: Re: [boost] Line endings fixes

Hi Vitaly,

back from vacation myself.

On Thursday, 1. August 2013 09:23:26 Vitaly Budovski wrote:

...

Hi Juergen,

I've had a look and the svn:eol-style property seems to be present in the patches. Perhaps I've missed a file or two?

yes, those are

<snip>

M libs/math/dot_net_example/distribution_explorer/Properties/app.manifest which ist missing both eol-style and mime-type

I've sure this will be OK.

And I would like the svn:mime-type of the .svg file to be something like "text/x-xml" instead of "text/plain"

If no-one objects, I can fix those and commit the the result.

I'm CC'ing John (for most files are Boost.Math documentation svgs) and Dave so this does not get lost.

I'm sure John will make a more authoritative reply, but I have a vague remembrance that "text/x-xml" caused trouble so I'd hold off making a change until he replies. Will this also mean a change to our SVN and GIT config files? We'd certainly like a ping when this is done to ensure that the docs will still build OK. Paul --- Paul A. Bristow, Prizet Farmhouse, Kendal LA8 8AB UK +44 1539 561830 07714330204 pbristow@hetp.u-net.com

Hi, Juergen, I've been pretty busy. Thanks for catching those omissions. On 7 August 2013 22:09, John Maddock wrote:

...

And I would like the svn:mime-type of the .svg file to be something like "text/x-xml" instead of "text/plain"

SVG's should all be "image/svg+xml"

John, as I outlined in the original post to the list, having them set to binary mime type causes issues due to the inconsistent line endings of some SVG files. While image/svg+xml is correct for serving the files in a webpage, it's not really a good choice for storing in a repository. Cheers, Vitaly

...

If no-one objects, I can fix those and commit the the result.

By all means go ahead on the Boost.Math stuff.

John.

Hi John, On Wednesday, 7. August 2013 16:04:31 John Maddock wrote:

...

John, as I outlined in the original post to the list, having them set to binary mime type causes issues due to the inconsistent line endings of some SVG files. While image/svg+xml is correct for serving the files in a webpage, it's not really a good choice for storing in a repository.

Sorry I missed the original message. I don't really care what it's set to as long as I can point the browser to: http://svn.boost.org/svn/boost/trunk/libs/math/doc/equations/airy.svg and see the image, not the XML.

Well, please take a look at the conversion result at https://github.com/boostorg/math/blob/master/doc/equations/airy.svg It seems that github delivers the source. You may want to open an extra thread for this,

Sorry, but what problem does mixed line endings in these actually cause?

Well, the files get converted to the platforms native line ending on git clone using the .gitattributes files or so. In the end, this leads to modified files immediately after the clone, which is undesirable.

They're intended to be viewed as images, they should never be edited as source as they're script generated.

Well, then why have they mixed eols? Can the script be fixed? And I guess I can edited them using inkscape ;-) Okay, I'll think about it. But the safest way would be to just correct the eols and leave the mime-type as-is. What happens to github is another story. Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! * voice: ++49 4257 300 ! Fährstraße 1 * fax : ++49 4257 300 ! 31609 Balge/Sebbenhausen * jhunold@gmx.eu ! Germany

...

Sorry I missed the original message. I don't really care what it's set to as long as I can point the browser to: http://svn.boost.org/svn/boost/trunk/libs/math/doc/equations/airy.svg and see the image, not the XML.

Well, please take a look at the conversion result at https://github.com/boostorg/math/blob/master/doc/equations/airy.svg

It seems that github delivers the source. You may want to open an extra thread for this,

Better is to ask github to show you it raw e.g. https://raw.github.com/boostorg/math/master/doc/equations/airy.svg. Note that github always serves everything as text/plain irrespective. This is apparently for security. Nothing we can do can change that behavior. In other words, you cannot use github as a content server, except as plain text. I might additionally add that git has no notion of storing MIME type anyway. It doesn't even store any difference between text or binary. All text vs binary differences happen at the point of conversion in and out of storage and so whatever gitattributes happens to say at the time is what takes effect.

...

Sorry, but what problem does mixed line endings in these actually cause?

Well, the files get converted to the platforms native line ending on git clone using the .gitattributes files or so. In the end, this leads to modified files immediately after the clone, which is undesirable.

Not exactly. Native EOL not matching what the git index says is in fact the problem.

...

They're intended to be viewed as images, they should never be edited as source as they're script generated.

Well, then why have they mixed eols? Can the script be fixed? And I guess I can edited them using inkscape ;-)

Okay, I'll think about it. But the safest way would be to just correct the eols and leave the mime-type as-is.

Dave Abrahams has asked me to look into fixing this mismatched EOL problem in the git mirrors, as it can't be a gitattributes problem seeing as that is correctly specified for .svg extensions. I will try to do so this weekend. Niall --- Opinions expressed here are my own and do not necessarily represent those of BlackBerry Inc.

...

Better is to ask github to show you it raw e.g. https://raw.github.com/boostorg/math/master/doc/equations/airy.svg.

Note that github always serves everything as text/plain irrespective. This is apparently for security. Nothing we can do can change that behavior. In other words, you cannot use github as a content server, except as plain text.

... except for github pages, which are designed for exactly this.

http://pages.github.com

Yeah, I always keep forgetting about those. If there ever was a way of not doing project page publishing, that is it. (What they really need is to let you do all that magic subbranch hassle server side, so basically I say I want directory X in the SCM published as HTML and they take care of it). Anyway, my point still stands: without magic subbranch hassle to bounce content through github pages, I know of no way to get github to serve SCM content as anything but text or binary blob. If you have a SVG file in the SCM, github will serve that as text, PNGs are offered for download, not rendered etc. Niall --- Opinions expressed here are my own and do not necessarily represent those of BlackBerry Inc.

...

...

They're intended to be viewed as images, they should never be edited as source as they're script generated.

Well, then why have they mixed eols? Can the script be fixed? And I guess I can edited them using inkscape ;-)

You could edit then in Inkscape, and I'm sure that would work just fine regardless of the line-endings they use.

The question is what should the script be fixed to? I guess all \n's and no \r's is safest?

There is something real weird going on, as supposedly git should rationalize mismatched EOLs during a svn commit replay irrespective of mismatch in the svn content. As I mentioned, I'll look into what could be causing the problem this weekend hopefully. Niall --- Opinions expressed here are my own and do not necessarily represent those of BlackBerry Inc.

...

There is something real weird going on, as supposedly git should rationalize mismatched EOLs during a svn commit replay irrespective of mismatch in the svn content.

If by “svn commit replay” you mean what Boost2Git is doing, then no, there’s no reason to think that should happen. We’re using git-fast-import, which just eats the raw bytes you send it and doesn’t have an opinion about whether they’re text or not.

That is extremely useful to know, and indeed I can see where the EOL problem might have come from now.

Also, let me be very clear, just in case there’s some misunderstanding about this: it is not Boost2Git’s job to fix bad SVN commits in the past. If something like that needs to be fixed before the transition, it should be fixed by making new commits to SVN, which means existing history stays as it is. Another option is to make fixes in Git immediately after the transition (which again, doesn’t change history). That means the only things we can alter that change the way history appears in Git are .gitattributes, repositories.txt, etc., and non-Boost-specific logic in the Boost2Git source code—at least, not without very extensive justification.

If it is the original SVN which is in error, it could be that SVN fixes it up on checkout and hence we've never noticed it before. That implies the need for a tool which can grok the SVN repo directly for mismatched EOLs. Niall --- Opinions expressed here are my own and do not necessarily represent those of BlackBerry Inc.

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Vitaly Budovski Sent: Saturday, August 10, 2013 5:57 AM To: boost@lists.boost.org Subject: Re: [boost] Line endings fixes

Hi,

On 10 August 2013 00:39, Niall Douglas wrote:

...

If it is the original SVN which is in error, it could be that SVN fixes it up on checkout and

hence we've never noticed it before. That implies the need for a tool which can grok the SVN repo directly for mismatched EOLs.

...

Niall

Niall, the issue is that SVN treats most SVG files as binary currently. As far as I'm aware, it does not fix anything up on checkout. If someone was to try and regenerate those particular files on a system with different line endings then SVN will flag those files as having been modified. The proper fix would be to correct the mime-type and line endings in the SVN properties. If a file needs to be rendered in a visual format on the webpage then the webserver should handle the appropriate mime-type setting.

The best option in my opinion would simply be not to store any generated files in the repository as they have the potential of getting out of sync with the source as well as bloating the size of the repository.

I'm confused, but does it help you to know that the many SVG files for Booth.Math are generated by a C++ program that we run. This program writes html and end of line as std::endl;. These SVG files are used to generate the PDF version of Boost.Math docs (because using PNG produces very poor display). These SVG files are also converted to PNG for the html version. (We'd like to use the SVG for PNG because they are much higher quality (and the conversion is a tiresome nuisance), but some browsers still in use don't support SVG :-( ) (Mentioning no names, but you can guess who I am getting at...) So eventually we'd like to be able to switch to displaying the SVG versions from html. HTH Paul --- Paul A. Bristow, Prizet Farmhouse, Kendal LA8 8AB UK +44 1539 561830 07714330204 pbristow@hetp.u-net.com

This isn't just a SVG problem: bootstrap.bat in the root directory has the wrong svn:eol-style for example!

Options:

1. If we don't care that checkouts from git of old revisions may have incorrect EOLs, we can commit a fix to all affected files in SVN HEAD to ensure their svn:eol-style exactly matches what https://svn.boost.org/trac/boost/wiki/BoostSubversion says it should be.

2. If we do care that checkouts from git any revision are always correct, we'll need to do some monkeypatching during the svn to git conversion

On 13 Aug 2013 01:46, "Niall Douglas" wrote: process

(i.e. check for incorrectness, and fix).

Niall

I prefer the first option, since it's not just a git problem and keeps the two repositories consistent. Might also be a good idea to introduce some pre-commit SVN hooks to ensure appropriate line endings.

-----Original Message----- From: Boost [mailto:boost-bounces@lists.boost.org] On Behalf Of Vitaly Budovski Sent: Monday, August 12, 2013 10:58 PM To: boost@lists.boost.org Subject: Re: [boost] Line endings fixes

...

This isn't just a SVG problem: bootstrap.bat in the root directory has the wrong svn:eol-style for example!

Options:

1. If we don't care that checkouts from git of old revisions may have incorrect EOLs, we can commit a fix to all affected files in SVN HEAD to ensure their svn:eol-style exactly matches what https://svn.boost.org/trac/boost/wiki/BoostSubversion says it should be.

2. If we do care that checkouts from git any revision are always correct, we'll need to do some monkeypatching during the svn to git conversion

On 13 Aug 2013 01:46, "Niall Douglas" wrote: process

...

(i.e. check for incorrectness, and fix).

Niall

I prefer the first option, since it's not just a git problem and keeps the two repositories consistent.

Might also be a good idea to introduce some pre-commit SVN hooks to ensure appropriate line endings.

I thought we had these already? - I've certainly fallen foul of some checks :-( Paul --- Paul A. Bristow, Prizet Farmhouse, Kendal LA8 8AB UK +44 1539 561830 07714330204 pbristow@hetp.u-net.com

On 13 August 2013 21:42, John Maddock wrote:

I just tried to do this and SVN wouldn't let me - the combination listed on that webpage for SVG files svn:eol-style=native; svn:mime-type=image/svg+xml isn't valid and SVN won't allow you to apply it to a file. So you can either have the correct mime-type, or the correct eol-style, but not both.

John.

eol-style can only be applied to text mime-type files. image/svg+xml is binary. You need to change the mime-type also. Cheers, Vitaly

...

1. If we don't care that checkouts from git of old revisions may have incorrect EOLs, we can commit a fix to all affected files in SVN HEAD to ensure their svn:eol-style exactly matches what https://svn.boost.org/trac/boost/wiki/BoostSubversion says it should be.

2. If we do care that checkouts from git any revision are always correct, we'll need to do some monkeypatching during the svn to git conversion process (i.e. check for incorrectness, and fix).

Niall

I prefer the first option, since it's not just a git problem and keeps the two repositories consistent.

Dave Abrahams and I have come to an agreement that fixing up all past git commits to be EOL correct is probably the one choice acceptable. We're currently working out the details off list.

Might also be a good idea to introduce some pre-commit SVN hooks to ensure appropriate line endings.

Boost already has those, but some incorrectness has slipped in over time. Niall --- Opinions expressed here are my own and do not necessarily represent those of BlackBerry Inc.

Hi John, On Wednesday, 7. August 2013 13:09:38 John Maddock wrote:

...

And I would like the svn:mime-type of the .svg file to be something like "text/x-xml" instead of "text/plain"

SVG's should all be "image/svg+xml"

Why? They are plain text files containing xml. Nothing binary in them which is non-text mime-type is designed for. I have inkscape and doxygen working with text/* without any problems on both Linux and Windows.

...

If no-one objects, I can fix those and commit the the result.

By all means go ahead on the Boost.Math stuff.

Okay, will fix the remaining files first. Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! * voice: ++49 4257 300 ! Fährstraße 1 * fax : ++49 4257 300 ! 31609 Balge/Sebbenhausen * jhunold@gmx.eu ! Germany

Hi, On 7 August 2013 23:02, Jürgen Hunold wrote:

Hi!

<snip>

This leaves the question on how to treat the .svg files.

Please remember that this a temporary issue as git does not have anything like svn properties.

On Wednesday, 7. August 2013 22:15:47 Vitaly Budovski wrote: <snip>

...

While image/svg+xml is correct for serving the files in a webpage, it's not really a good choice for storing in a repository.

If this change breaks doc generation or the webpage due to the mime-type change, they will definetely have to be adapted after the git transition.

That depends on how Apache is configured. I think it would be better to let Apache handle the mime-types on files where specific rendering is required (i.e. SVG) http://serverfault.com/questions/451500/is-it-possible-for-the-subversion-ap... Cheers, Vitaly

Yours,

Jürgen --