Git hardlinks, and developing
OK so I've just been hit by my first Git SNAFU. 1) I checked out modular Boost as per https://svn.boost.org/trac/boost/wiki/TryModBoost 2) Changed libs/multiprecision to the develop branch also as per https://svn.boost.org/trac/boost/wiki/TryModBoost 3) Added some pending changes from the old SVN Trunk that I didn't commit before the changeover. 4) Ran the tests and everything failed, even though it was passing on SVN :-( The issue was that the headers under boost/ were now *copies* of the last release and no longer pointers to the new "develop" code. Interestingly some of the hard links were updated when I ran bjam, but apparently not all :-( Running bjam -a fixes the issue, but it's a very easy trap for the unwary.... John. PS, one other very annoying issue for VC++ users: 1) Open a test file. 2) Select a #included boost file, right click and select "Open document". 3) Document opens, but gives you a copy under boost/ not the "real" header under libs/mylib/. 4) Run away screaming after you've spent an XXX hours editing the wrong file....
On 4 December 2013 13:40, John Maddock <john@johnmaddock.co.uk> wrote:
OK so I've just been hit by my first Git SNAFU.
1) I checked out modular Boost as per https://svn.boost.org/trac/boost/wiki/TryModBoost 2) Changed libs/multiprecision to the develop branch also as per https://svn.boost.org/trac/boost/wiki/TryModBoost 3) Added some pending changes from the old SVN Trunk that I didn't commit before the changeover. 4) Ran the tests and everything failed, even though it was passing on SVN :-(
The issue was that the headers under boost/ were now *copies* of the last release and no longer pointers to the new "develop" code. Interestingly some of the hard links were updated when I ran bjam, but apparently not all :-(
On the "non-linked headers" thread there's discussion about using soft links instead of hard links, which will hopefully fix this.
1) I checked out modular Boost as per https://svn.boost.org/trac/boost/wiki/TryModBoost 2) Changed libs/multiprecision to the develop branch also as per https://svn.boost.org/trac/boost/wiki/TryModBoost 3) Added some pending changes from the old SVN Trunk that I didn't commit before the changeover. 4) Ran the tests and everything failed, even though it was passing on SVN :-(
The issue was that the headers under boost/ were now *copies* of the last release and no longer pointers to the new "develop" code. Interestingly some of the hard links were updated when I ran bjam, but apparently not all :-(
On the "non-linked headers" thread there's discussion about using soft links instead of hard links, which will hopefully fix this.
I saw that. However, my Vista machine is apparently reporting that soft links are *not* supported :-( Of course it's about time it was junked in favour of something newer, but that's a whole other issue! John.
AMDG On 12/04/2013 08:01 AM, John Maddock wrote:
On the "non-linked headers" thread there's discussion about using soft links instead of hard links, which will hopefully fix this.
I saw that. However, my Vista machine is apparently reporting that soft links are *not* supported :-(
Of course it's about time it was junked in favour of something newer, but that's a whole other issue!
I think it's a permissions issue. In Windows 7, (I don't know about Vista) only admins can create symbolic links by default. It looks like we can use a junction (mklink /J) instead for directories. In Christ, Steven Watanabe
1) I checked out modular Boost as per https://svn.boost.org/trac/boost/wiki/TryModBoost 2) Changed libs/multiprecision to the develop branch also as per https://svn.boost.org/trac/boost/wiki/TryModBoost 3) Added some pending changes from the old SVN Trunk that I didn't commit before the changeover. 4) Ran the tests and everything failed, even though it was passing on SVN :-(
The issue was that the headers under boost/ were now *copies* of the last release and no longer pointers to the new "develop" code. Interestingly some of the hard links were updated when I ran bjam, but apparently not all :-(
Running bjam -a fixes the issue, but it's a very easy trap for the unwary.... John.
And another thing.... Ran "bjam -a headers" from the root directory to force the headers to be rebuilt, and the tests now build and pass OK... except that's a mistake because I see that old test results that depend on those headers were *not* rebuilt. So you need to bjam -a from the libs/mylib/test directory as well whenever you change branches :-( John.
AMDG On 12/04/2013 08:04 AM, John Maddock wrote:
1) I checked out modular Boost as per https://svn.boost.org/trac/boost/wiki/TryModBoost 2) Changed libs/multiprecision to the develop branch also as per https://svn.boost.org/trac/boost/wiki/TryModBoost 3) Added some pending changes from the old SVN Trunk that I didn't commit before the changeover. 4) Ran the tests and everything failed, even though it was passing on SVN :-(
The issue was that the headers under boost/ were now *copies* of the last release and no longer pointers to the new "develop" code. Interestingly some of the hard links were updated when I ran bjam, but apparently not all :-(
Running bjam -a fixes the issue, but it's a very easy trap for the unwary.... John.
And another thing....
Ran "bjam -a headers" from the root directory to force the headers to be rebuilt, and the tests now build and pass OK... except that's a mistake because I see that old test results that depend on those headers were *not* rebuilt. So you need to bjam -a from the libs/mylib/test directory as well whenever you change branches :-(
The reason that this is failing is probably that the timestamps on some of the headers that you copied in were older than the timestamps on the original. In Christ, Steven Watanabe
On 12/04/2013 05:48 PM, Steven Watanabe wrote:
AMDG
On 12/04/2013 08:04 AM, John Maddock wrote:
1) I checked out modular Boost as per https://svn.boost.org/trac/boost/wiki/TryModBoost 2) Changed libs/multiprecision to the develop branch also as per https://svn.boost.org/trac/boost/wiki/TryModBoost 3) Added some pending changes from the old SVN Trunk that I didn't commit before the changeover. 4) Ran the tests and everything failed, even though it was passing on SVN :-(
The issue was that the headers under boost/ were now *copies* of the last release and no longer pointers to the new "develop" code. Interestingly some of the hard links were updated when I ran bjam, but apparently not all :-(
Running bjam -a fixes the issue, but it's a very easy trap for the unwary.... John.
And another thing....
Ran "bjam -a headers" from the root directory to force the headers to be rebuilt, and the tests now build and pass OK... except that's a mistake because I see that old test results that depend on those headers were *not* rebuilt. So you need to bjam -a from the libs/mylib/test directory as well whenever you change branches :-(
The reason that this is failing is probably that the timestamps on some of the headers that you copied in were older than the timestamps on the original.
Some thoughts: Assuming we have changed b2 headers to prefer symlink for individual files: Then, in the cases where hardlinks are still used by b2 headers, does it not make sense for b2 headers to always, unconditionally fix all hardlinks -- all the time...? Maybe even a post checkout/reset hook in git where the b2 headers command is allowed to fail with a warning only. On platforms where copy is still used by b2 headers, forced unconditional copy would trigger redundant rebuild and would make no sense. However with a content change detection based condition for the copy, forced copy may make a lot of sense. Maybe even better if it is possible to also set filetime on the copy to the same as the original. I do not think we should attempt to do this in both directions, as it complicates too much to be worth it -- so this would not fix all potential issues with copies, but maybe the more surprising ones that are not self inflicted by editing files directly in the boost folders. -- Bjørn
On Wed, Dec 4, 2013 at 3:12 PM, Bjørn Roald <bjorn@4roald.org> wrote: Some thoughts:
Assuming we have changed b2 headers to prefer symlink for individual files:
Before we change anything I'd prefer to see a set of requirements developed. What scenarios do we expect to work out-of-the-box on what platforms? I'd like to see highest priority given to recent versions of the most common platforms used by Boost developer. I.E. Linux, Mac OS X, Windows. And getting the scenarios to work on recent versions before worrying about versions already abandoned by their vendors. Just my opinion, of course. --Beman
On 12/4/2013 4:13 PM, Beman Dawes wrote:
On Wed, Dec 4, 2013 at 3:12 PM, Bjørn Roald <bjorn@4roald.org> wrote:
Some thoughts:
Assuming we have changed b2 headers to prefer symlink for individual files:
Before we change anything I'd prefer to see a set of requirements developed. What scenarios do we expect to work out-of-the-box on what platforms?
On Linux symbolic links work on all Linux distributions. This is clearly the way to go on Linux for every link, whether directory or file, created within the boost/ subpath. We have already discussed why hardlinks are the poorer choice for files. I know nothing about the Mac. On Windows: 1) Symbolic links work for Windows Vista and higher if the user has administrative priveleges. 2) Junctions are a form of symbolic links for directories that cannot span network paths, which will not be a problem for modular boost organization. Junctions work on all versions of Windows from Windows 2000 on up and do not require any special priveleges. On Windows Vista and higher we should prefer symbolic links for the directories and files in the boost/ subpath if the user has administrative priveleges. Without administrator priveleges on Windows Vista and higher, or in Windows XP and lower, we should use junctions for directory links and hardlinks for the file links. Ideally in the modular boost future if we could limit all link in the boost/ subpath to directory links, we would be better off on Windows since hardlinks need never be used.
I'd like to see highest priority given to recent versions of the most common platforms used by Boost developer. I.E. Linux, Mac OS X, Windows. And getting the scenarios to work on recent versions before worrying about versions already abandoned by their vendors. Just my opinion, of course.
Understood.
On 4 Dec 2013 at 17:13, Edward Diener wrote:
1) Symbolic links work for Windows Vista and higher if the user has administrative priveleges.
Incorrect. You can create them if the calling uid has the SeCreateSymbolicLinkPrivilege. Granting this to all logged in users in the system is trivially achieved, and is one of the first things I do on any new Windows install.
On Windows Vista and higher we should prefer symbolic links for the directories and files in the boost/ subpath if the user has administrative priveleges. Without administrator priveleges on Windows Vista and higher, or in Windows XP and lower, we should use junctions for directory links and hardlinks for the file links.
A simple program can add SeCreateSymbolicLinkPrivilege for the Users group if not already present, or just for the local user for the paranoid. Then you can work with symlinks on Windows as if you were on POSIX (apart from the occasional differences in symlink semantics of course). Needing admin privs is of course a problem for users without access to admin privs e.g. in corporations. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
On 12/04/2013 08:09 PM, Niall Douglas wrote:
On 4 Dec 2013 at 17:13, Edward Diener wrote:
1) Symbolic links work for Windows Vista and higher if the user has administrative priveleges.
Incorrect. You can create them if the calling uid has the SeCreateSymbolicLinkPrivilege. Granting this to all logged in users in the system is trivially achieved, and is one of the first things I do on any new Windows install.
Do you need admin privileges to grant SeCreateSymbolicLinkPrivilege to users? -- Bjørn
On 5 Dec 2013 at 1:14, Bjørn Roald wrote:
Incorrect. You can create them if the calling uid has the SeCreateSymbolicLinkPrivilege. Granting this to all logged in users in the system is trivially achieved, and is one of the first things I do on any new Windows install.
Do you need admin privileges to grant SeCreateSymbolicLinkPrivilege to users?
You need admin privs to change privilege assignments, yes. It's a once off though. Note that proposed Boost.AFIO fully understands symlinks on both Windows and POSIX, and it's written using the NT kernel API instead of Win32 so it'll be much more portable across Windows implementations. If you need appropriate symlink handling code for Windows, AFIO is good place to mine it from. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
On 5/12/2013 11:13, Quoth Edward Diener:
1) Symbolic links work for Windows Vista and higher if the user has administrative priveleges. 2) Junctions are a form of symbolic links for directories that cannot span network paths, which will not be a problem for modular boost organization. Junctions work on all versions of Windows from Windows 2000 on up and do not require any special priveleges.
I commonly build boost from a shared network drive. I would be surprised if this supported either option. (This is at "boost user with tarball" level though, not "boost developer with git", so I don't know if that's affected by this issue.)
On 12/04/2013 10:13 PM, Beman Dawes wrote:
On Wed, Dec 4, 2013 at 3:12 PM, Bjørn Roald <bjorn@4roald.org> wrote:
Some thoughts:
Assuming we have changed b2 headers to prefer symlink for individual files:
Before we change anything I'd prefer to see a set of requirements developed. What scenarios do we expect to work out-of-the-box on what platforms?
I'd like to see highest priority given to recent versions of the most common platforms used by Boost developer. I.E. Linux, Mac OS X, Windows. And getting the scenarios to work on recent versions before worrying about versions already abandoned by their vendors. Just my opinion, of course.
OK, I give it a try. The list below is sort of in my priority order. The words /shall/ is stronger than /should/, and /may/ is optional: For simplicity I call file operated on when accessed from within the BOOST_ROOT/boost folder a *proxy header* or simply *proxy* of the *original* header, i.e.; all the "b2 headers" build target is doing is staging a bunch of proxy headers. Each requirement is followed by a compliance section with my understanding of current and potential compliance. All potential compliance require some work. Please comment! Requirement: BPH-01 Essential Calling "b2 headers" explicitly shall ensure all proxy headers have identical content of respective originals. Compliance: b2 headers current potential symlink folder: YES YES symlink proxy : YES YES hardlink proxy : YES/NO(1) YES(5) copy proxy : YES/NO(2) YES(6) Requirement: BPH--02 Desired - high priority Accidental (or casual) editing of proxy header files in boost folder should have the immediate effect of updating the respective original accordingly. Compliance: b2 headers current potential symlink folder: YES YES symlink file : YES YES hardlink file : YES/NO(1)(3) YES/NO(1) copy of file : NO NO Requirement: BPH-03 Desirable Changes to original header file should have the effect of updating the respective proxy immediately. Compliance: current potential symlink folder: YES(4) YES symlink file : YES(4) YES hardlink file : YES(1)(4) YES/NO(3) copy of file : NO NO Requirement: BPH-04 Desirable (short term) / Essential (longer term) Any build with b2 that depend on a headers in boost shall ensure those headers have identical content of respective originals in libs/*/include/boost folder. Note: longer term goal to get rid of need to do explicit b2 headers build Compliance: b2 headers current potential symlink folder: NO(3) YES symlink proxy : NO(3) YES hardlink proxy : NO(1)(3) YES copy proxy : NO(2)(3) YES Requirement: BPH-05 Desired When git commands change content in working directory, hooks may be used to call b2 headers. Compliance: potentially YES, if b2 headers is called by git hook, then compliance would be as in BPH-01. Requirement: BPH-06 Desired / Optional Building with b2 may remove orphan headers in boost folder. Note: Alternatives such as b2 clean-headers may be good enough and more feasible. Compliance: b2 headers current potential NO YES Requirement: BPH-06 Desirable b2 builds may warn the user if the system does not support robust features to manage header proxies and point to a web page with advise. E.g.: if b2 headers have to use hard links or copies. Compliance: current potential NO YES End-notes: (1) Git and potentially some other editors and tools may change one end in hardlink so it will break automatically reflective update of the other end. E.g.: deleting the link, then create a new file in its place, thus causing the other end to become a regular copy of the original file. (2) If file time of libs/*/include/boost header is older than proxy header, then current b2 header solution will fail. (3) b2 headers does currently not have a complete dependency graph with productions of individual proxies that depend their respective originals. (4) b2 headers must have been run once to set up symlink proxies (5) b2 may always replace all hardlink proxies, thus fixing #(1) issues from the original side, e.g. git braking hardlinks. (6) b2 may always copy original files to copy proxies if b2 detect file content mismatch. I think changing b2 headers preference for proxy methods is the best step 1, given also that symlink is more or less universally available on newer developer host OS'es, given some teaks may be needed on Windows for privileges. I suggest changing current: 1. symlink dir 2. hardlink file 3. symlink file 4. copy preference to: 1. symlink dir 2. symlink junktion dir 3. symlink file 4. hardlink file 5. copy That would take care of BHP-01, BHP-02, BHP-03, for most users. Then I would actually do BHP-06, as it is simple and would hopefully warn those that are not as lucky, and potentially help them fix it or understand and be cautious, -- Bjørn -- Bjørn
participants (8)
-
Beman Dawes -
Bjørn Roald -
Daniel James -
Edward Diener -
Gavin Lambert -
John Maddock -
Niall Douglas -
Steven Watanabe