big problem with dependency changes
Currently the master branch has a few problems. Here is one typical example. My project uses the current master to provide boost libraries which I'm not working on. One file has the statement #include <boost/utility/enable_if.hpp> which fails to compile since boost/utility/enable_if.hpp has been moved to core. This raises a few questions. a) Isn't the master branch always supposed to be working? b) There are a lot of intermediate checkins which break the master. Should these be done an a branch and, once they are believed to be all correct, merged in as a group. This would support the concept a) above. c) will I have to change code in the library I maintain to manually change all occurrences of #include <boost/utility/enable_if.hpp> to #include <boost/core/enable_if.hpp> ? d) Why are the tests on the master branch passing? Robert Ramey -- View this message in context: http://boost.2283326.n4.nabble.com/big-problem-with-dependency-changes-tp466... Sent from the Boost - Dev mailing list archive at Nabble.com.
On 06/16/2014 12:11 PM, Peter Dimov wrote:
For those of you that struggle with this when you use b2 on your system, You should get a line at the beginning of the output saying: - symlinks supported : yes If not, b2 is not able to create symbolic links for individual files on your system. There may be a number of reasons for this, but on recent MS Windows systems it is usually related to user privileges, the tool used to create symbolic links require elevated privileges. If b2 are not able to create symbolic links for "b2 headers" targets, it will use hard links or file copies. Both of these alternatives to symbolic links have severe issues that could cause you to end up with stale files in the boost folder, or cause you to edit files that are later overwritten by b2. For enabling b2 use of symbolic links on windows, have a look at the following information from the web (quoting from my posting 2 February on the same issue.): refs: http://stackoverflow.com/questions/15320550/secreatesymboliclinkprivilege-ig... http://msdn.microsoft.com/en-us/library/bb530410.aspx http://superuser.com/questions/148937/windows-server-2008-create-symbolic-li... http://superuser.com/questions/124679/how-do-i-create-a-link-in-windows-7-ho... Most promising may be: Provide SeCreateSymbolicLinkPrivilege privilege to the user who is running the script. If the user is Administrator on the machine you may need to turn off UAC. Note I have not tested any of this as I have no problems on my systems. So please give feedback to the list if any of this work for you. Thanks. -- Bjørn
On 06/16/2014 07:55 PM, Peter Dimov wrote:
OK, junctions are used for the directories on Windows if symlinks are not working.
and it works reasonably well. Perhaps I'm not stressing it enough though.
perhaps, but it could also be something else. I you do not mind, could you try in a clone for used testing only, or at least not where you have work you have not committed: rm boost/filesystem.h b2 headers cd libs/filesystem git reset --hard HEAD then edit boost/filesystem.h or libs/filesystem/include/boost/filesystem.h Then compare them, if they are the same, then I am very interested in understanding why. -- Bjørn
Bjørn Roald wrote:
The files are different after I edit one of the two hard links. When I edit in boost/ - which I never do - when I run the tests, b2 then sees the file as modified, but does not update the link in boost/ (the link being newer than the source in include/.) When I edit in include/ - which is what I do - b2 sees the file as modified and updates the link in boost/ to point to it. Moral of the story: don't edit the files in boost/. The exceptions are the headers which b2 can't see as dependencies - those included via a macro. They never get their links updated unless one runs "b2 headers". We've fixed some of them, but not all.
On 06/17/2014 10:46 AM, Peter Dimov wrote:
OK, that is what I would expect. Thank you for testing,
Good! but even if all are fixed, it is not good to have these subtle issues where users have to be careful. Part of the problem is that many tools such as IDE's with built-in symbol browsing, debugers etc. will lead users to those headers they should not edit. -- Bjørn
On 18/06/2014 09:29, Bjørn Roald wrote:
Perhaps b2 could check both timestamps are equal and sync bidirectionally? Copied or hardlinked files should have the same timestamp (not sure about symlinks, but they should probably be ignored anyway since they won't have this issue), and both manually edited or freshly git-updated files should have a newer timestamp than their counterpart (modulo possible 2 second filesystem resolution). The only thing I can envisage possibly tripping this up is if someone untars/unzips a fresh release build over the top of their existing tree, which could potentially have older timestamps than the counterpart files, if the user has been editing them. (Especially if they're using the same release build as previous, to try to revert to originals.) But that seems like a case in which users ought to have cleaned the tree first in any event, and shouldn't be an issue for upgrades, as those have different dir names embedded in the archive. Although I suspect most users using release builds will not run b2 more than once anyway.
To me, this seems a *big* problem :-( I'd fail the whole idea of using hard or symlinks without some method of protecting the poor unsuspecting hapless users from this elephant trap. Been there, fell in ! Paul --- Paul A. Bristow Prizet Farmhouse Kendal UK LA8 8AB +44 01539 561830
On 06/18/2014 10:51 AM, Paul A. Bristow wrote:
Agree, I for one do not want to down-grade the severity of this.
I'd fail the whole idea of using hard or symlinks without some method of protecting the poor unsuspecting hapless users from this elephant trap.
I think symlinks will protect you just fine, and it is supported by b2. So it is not bad at all for users where b2 can use symlinks :-) The problem is to enable symlink support in a simple robust way that does not sacrifice security for users that dont have this by default. On Windows I fail to find a way. I think solving that should be a primary goal now, but it seems we need more detailed understanding of Windows privileges and UAC than I posses. I wonder: Could we possibly write or download an executable that work around this UAC strap-down and create symbolic links for Windows users in the Administrator group that has the SeCreateSymbolicLinkPrivilege set. If that is possible, then a sensible workaround would be feasible. I worry it is not possible, but I will be happy if I am wrong. Down the road there are other options than staging of headers in the legacy monolithic boost folder structure. This may be achieved relatively simply by module level dependencies and multiple -I statements to the compiler. But this approach has its own set of disadvantages and a number of boost tools are not ready for it. There are probably a few pitfalls as well with dubious workarounds, e.g. command line limitations on some systems. End users will likely always be better off with a deployed set of selected boost modules with a single boost header file structure. So that will be simpler with a deployment system that is not using the git repositories. But as some pointed out, they are expected to download, possibly build boost once, then use. So as long as they don't overwrite their copy of an old release, there is no problem. If they do, then it is reasonable to require a b2 clean or similar. -- Bjørn
On 19/06/2014 04:56, Bjørn Roald wrote:
What about two-way sync, as I suggested earlier?
I think the only way that could work is if a system-level (and thus admin-permissions) service were installed that accepted requests from non-admin users to create symlinks on their behalf. (I hope the security pitfalls of such a thing are obvious to everyone.) It actually gets weirder than that, though. It's possible for a user with administrator rights to enable the symlink creation privilege for all users. After doing that, though, if UAC is still enabled then while any limited user would be able to create symlinks, the admin user will still only be able to do so from an elevated process. (This is because this is one of the privileges that is explicitly denied to the restricted token created for non-elevated-admin-user processes -- and as far as I'm aware there isn't a setting for *that*.) And yes, that's very weird. (Though it's true that symbolic links can be dangerous things, and it's true that most Windows apps aren't expecting them, which can make them even more dangerous. Traversing through symlinks to delete subfolders will ruin many a day.)
Another possible pitfall is that it may delay detection of cases when different libraries clobber each others' header files (although the naming policies and directory structures should limit this).
Gavin Lambert wrote:
What about two-way sync, as I suggested earlier?
Two way sync seems somewhat dangerous. Perhaps b2 aborting with an error if it sees a newer file in boost/ will be enough. On second thought, this won't work when the same header is present in more than one module. This happens when a header is being moved and is added to its new module before being removed from its old place. Currently b2 doesn't mind, it just updates the link twice and the later copy wins.
I agree that this is most desirable - or even essential?
Another complexity I fear is that requiring use of Admin privileges is that it mean users cannot use tools like Tortoise GIT but must use a GIT command line from an elevated privs GIT Bash session. Enforcing command-line only really isn't acceptable, even for library authors. Or is it only running b2 headers that is the problem? Paul --- Paul A. Bristow Prizet Farmhouse Kendal UK LA8 8AB +44 01539 561830
Or is it only running b2 headers that is the problem?
Maybe I have missed it in the previous conversations, but why have ./b2 headers make the /boost folder at all? When using boost-build, the headers target can add all the include paths for the various projects. If working on a project not using boost-build, generally I would install boost (at the very least into a folder). I ask, how are people depending on the generated /boost directory? Is it only to build the boost libraries? Regards, Thomas
On 06/20/2014 08:55 AM, Vladimir Prus wrote:
For info: CMake solves this on Windows (at least for linking) by creating a file with commandline options, which can be passed to cl command-line as a single filename option. But this solution is a real pain as these temporary files are gone when you want to see what are passed to the compiler. So there is no simple way to trace what is going on when something is not working. Tracing down some problems in cmake on windows has been a real pain due to this. So, in my opinion, if we can manage to get it right, staging the headers as b2 headers do is so much simpler to reason about. Besides that, the stage structure is probably how end users will see boost headers. So the developers get to eat their own kake, which is a good thing. -- Bjørn
Bjørn Roald wrote:
FYI: http://cmake.org/gitweb?p=cmake.git;a=blob;f=Modules/Platform/Windows.cmake;... Thanks, Steve.
On 06/21/2014 12:59 PM, Stephen Kelly wrote:
OK, I see. It would have been nice if it just set CMAKE_START_TEMP_FILE to "" for me when I pass VERBOSE=1 to nmake. But that may re-create the possible issues with the command line length that use of temp files is designed to work around. The best thing to do for build systems using such temp files may be simply to list the content of the file after the command line when doing verbose build output. thanks, -- Bjørn
On 06/16/2014 07:42 PM, Paul A. Bristow wrote:
Well, it does under certain conditions: A boost/foo/bar.h hard link may end up as a copy of a deleted file that used to be at libs/foo/include/boost/foo/bar.h. A boost/foo/bar.h hard link may also end up as a copy of an old version of a changed file that used to be at libs/foo/include/boost/foo/bar.h. Basically all regular files are hard links to the file content. The hard link in the boost/foo/bar.h and its original are equivalent references to the same file content. They have shared ownership to the file (aka. shared_ptr). If the original is deleted, well, the file lives on as the other hard link lives on. If the original is changed by first deleting, then write to a new file with same name, then the boost/foo/bar.h hard link point to the old version of libs/foo/include/boost/foo/bar.h. Regrettably more tools than I expected do the later type of delete then write sequence rather than in-place overwrite when editing files. Git being one offender :-( Symlinks are just forwarding one name to the target file path. Thus neighter of the above problems causes real issues. You may still have stale symlinks to deleted files, but accessing them would fail when attempted as the original can not be found.
Don’t'we need to change the instructions *always* "run from administrator"?
I don't know. Some users are concerned about running as Administrator, and I am not sure I would recommend that at all. Some users may not be allowed to be Administrators in a corporate environment. The problem seems to be that even if you are in the Administrator group, you can not do this as your normal user in scripts. You have to elevate the script, which I assume means you run everything in the script as the Administrator user. That may be a poor idea. Simply setting the SeCreateSymbolicLinkPrivilege for the user seems more promising as general guideline, But there are two issues with that as well: 1. You have to be Administrator to set privelages, or you need help from somebody that is. I would not think this is a problem for most boosters. To change the policies: - Launch secpol.msc via Start or Start → Run. - Open Security Settings → Local Policies → User Rights Assignment. - In the list, find the "Create symbolic links" item, which represents SeCreateSymbolicLinkPrivilege. - Double-click on the item and add yourself (or the whole Users group) to the list. The changes will apply when you log out and log in again. 2. If you are in the Administrators group, you also need to disable User Access Control(UAC). I have no idea if that is a good idea. As this only apply to members of the Administrator group, I guess using a separate user account could be an alternative work around if you or your sysadmin are concerned about turning UAC off. To turn UAC off in Vista/Win7/8/8.1 see here: http://www.howtogeek.com/howto/windows-vista/disable-user-account-control-ua... -- Bjørn
On 17/06/2014 08:12, Bjørn Roald wrote:
It's likely to be defensive programming. Writing to a temporary file with an alternate name and then executing a delete-rename or rename-rename-delete sequence is generally as close as you can get to an atomic update that doesn't leave behind partial files in case of failure (thereby possibly destroying content), at least without "real" transaction support (which does exist but is rarely used in Windows). (This is because writes are slow, but deletes and renames are nearly instant and are less likely to fail.) Thus you'll find version-control systems and other software designed for reliability will do that more often than not, although it's more hit-n-miss with general-purpose applications (eg. editors). And most Windows software is not written to be symlink-aware because they're very rare.
In general, it is not a good idea. While Linux users (and more importantly, applications) are used to the idea of logging in to a completely separate account for day-to-day vs. admin operations, Windows users (and applications) generally expect to use the same account for both. UAC allows the distinction between elevated or not, which lets the non-admin tasks be run more safely (like Linux). While it's possible for a sufficiently disciplined user to run dual accounts like on Linux, this often confuses applications in unexpected ways.
On 06/17/2014 03:04 AM, Gavin Lambert wrote:
Yes you are right, have done that many times in my own code.
In fact it was a discussion on the git developer list on the issue, and they decided to ignore the fact that they knew they would break hard links. I think the rationale was partly performance and partly the fact that it would not help much anyway. Other tools would break hardlinks and the reliance on hard links would be broken anyway.
And most Windows software is not written to be symlink-aware because they're very rare.
Did you mean hard-link aware? In my mind, whether they are symlink aware or not would not make much of a difference.
Yes, I would not like to give the general advice to turn UAC off, what people elect to do is their own business. But this leaves us with a problem for "b2 headers" on Windows, as we have no clear advice of how to deal with lack of symlink support that is simple, easy and secure. At least unless you are happy about not being in the Administrators group. The situation can be improved by making the "b2 header" targets proper targets in b2. For performance reasons, a shortcut is used currently I think. I believe that is also the reason you sometimes need to call "b2 headers" explicitly. Further b2 could detect hardlinks that has become copies in the boost folder. Although this would be an improvement, and it would fix build issues, it is not close to as good as symbolic links. Orphaned hardlinks to deletad header files would still be a potential problem. Also, other tools and the user will still suffer. There is a window from the hardlink relation is broken to next time you build where bad things may happen. E.g. you edit in the wrong file, and then your work is overwritten. Still looking for proper user level symlink support on windows. -- Bjørn
On Sun, Jun 15, 2014 at 10:23 PM, Robert Ramey wrote:
The code lives in the Core module, but the Core module provides <boost/utility/enable_if.hpp> which just forwards to <boost/core/enable_if.hpp> so your code should not fail to compile.
No. There's no reason why Core's <boost/utility/enable_if.hpp> cannot live on, just marked as deprecated, with all documentation referring to <boost/core/enable_if.hpp>.
d) Why are the tests on the master branch passing?
Did you run git submodule update --init? Glen
participants (11)
-
Andrey Semashev -
Bjørn Roald -
Eric Niebler -
Gavin Lambert -
Glen Fernandes -
Paul A. Bristow -
Peter Dimov -
Robert Ramey -
Stephen Kelly -
Thomas Suckow -
Vladimir Prus