AMDG On 03/13/2015 04:01 PM, Andrey Semashev wrote:

On Saturday 14 March 2015 00:29:56 you wrote:

...

Which makes me wonder. (a) When will MS get around to fix this in Windows? (I suspect "never" is the answer). (b) AFAIK, there is Windows API that can work with long paths (32767 chars or something?). Can bjam be updated to use these API to work around the problem?

It probably can. It means more platform specific code, though, as b2 currently just calls fopen.

(c) As an alternative, bjam could have an option to compress the path to a shorter hash value.

b2 --hash (reduces the path to an md5 hash) b2 --abbreviate-paths (shorter but still somewhat human readable) In Christ, Steven Watanabe

Le 13/03/15 23:18, Steven Watanabe a écrit :

AMDG

On 03/13/2015 04:01 PM, Andrey Semashev wrote:

...

On Saturday 14 March 2015 00:29:56 you wrote:

...

Which makes me wonder. (a) When will MS get around to fix this in Windows? (I suspect "never" is the answer). (b) AFAIK, there is Windows API that can work with long paths (32767 chars or something?). Can bjam be updated to use these API to work around the problem?

It probably can. It means more platform specific code, though, as b2 currently just calls fopen.

Hi, fopen can also work with UNC path, which does not have the MAX_PATH=255 limitation: - https://msdn.microsoft.com/en-us/library/yeby3zcb.aspx - https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%... A possible workaround would be to prefix the path name with "\\?\" then. If there is an intermediate cmd.exe, I suspect passing filename with the previous prefix works as well. How to debug at which point things start failing? Best, Raffi

Le 19/03/15 16:22, Steven Watanabe a écrit :

AMDG

On 03/19/2015 08:27 AM, Raffi Enficiaud wrote:

...

fopen can also work with UNC path, which does not have the MAX_PATH=255 limitation: - https://msdn.microsoft.com/en-us/library/yeby3zcb.aspx - https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%...

A possible workaround would be to prefix the path name with "\\?\" then.

It's a bit more complicated then that, since \\?\ requires an absolute path. Also, I thought that you need to use wchar_t as well.

Right, I made an attempt and most of the files are in relative path. Is there any function in C retrieving the full path? Also, I think that the full chain should be in full path as well (eg mkdir in common.jam should prefix the absolute path with \\?\ to work).

...

If there is an intermediate cmd.exe, I suspect passing filename with the previous prefix works as well.

How to debug at which point things start failing?

This particular failure happens towards the end of function.c. You can find it by searching for fopen.

Le 13/03/15 23:18, Steven Watanabe a écrit :

It probably can. It means more platform specific code, though, as b2 currently just calls fopen.

...

(c) As an alternative, bjam could have an option to compress the path to a shorter hash value.

b2 --hash (reduces the path to an md5 hash) b2 --abbreviate-paths (shorter but still somewhat human readable)

Possible alternative: mapping the folder containing boost to a drive, with subst, as a precommand of the build: http://superuser.com/questions/644684/mapping-drive-letters-to-local-folders It would be better if we can run bjam without this limitation. I propose to move any further discussion to the boost.build list. Best, Raffi

AMDG On 03/13/2015 06:26 PM, Niall Douglas wrote:

On 14 Mar 2015 at 0:29, Andrey Semashev wrote:

...

(b) AFAIK, there is Windows API that can work with long paths (32767 chars or something?). Can bjam be updated to use these API to work around the problem?

Proposed Boost.AFIO has always used the NT kernel API directly, and <snip>

What AFIO does is completely irrelevant, since b2 is written in C and cannot use Boost anyway. In Christ, Steven Watanabe

On 3/14/2015 12:49 PM, Niall Douglas wrote:

On 13 Mar 2015 at 19:47, Steven Watanabe wrote:

...

...

...

(b) AFAIK, there is Windows API that can work with long paths (32767 chars or something?). Can bjam be updated to use these API to work around the problem?

Proposed Boost.AFIO has always used the NT kernel API directly, and <snip>

What AFIO does is completely irrelevant, since b2 is written in C and cannot use Boost anyway.

I didn't know that b2 is written in C. However, there is as it happens relevance - AFIO's stable DLL export ABI is intentionally designed to be C compatible such that a C API can be provided in addition to the metaprogrammed C++ one.

I just need to get round to writing the libclang AST to C bindings generator, that's all. Time, as always.

I believe Steve meant that b2 uses the fopen/fread/fwrite series of C standard library functionality for cross-platform purposes and it is this which imposes the limit to the filespec path. Obviously b2 could try to use the same NT kernel API which AFIO uses to get around the 256 character limitation if cross-platform programming were not the issue.

AMDG On 03/14/2015 12:13 PM, John Maddock wrote:

...

I believe Steve meant that b2 uses the fopen/fread/fwrite series of C standard library functionality for cross-platform purposes and it is this which imposes the limit to the filespec path. Obviously b2 could try to use the same NT kernel API which AFIO uses to get around the 256 character limitation if cross-platform programming were not the issue.

I suspect it's more pernicious than that: bjam surely invokes the system's shell to execute commands, and it's there in cmd.exe that we're hitting the issues?

It's quite possible that we'd hit problems in cmd.exe, but it currently manifests earlier when writing the .rsp file, which b2 does directly. In Christ, Steven Watanabe

On Fri, Mar 13, 2015 at 8:26 PM, Niall Douglas wrote: ... deleting a directory tree

which will randomly fail on Windows if you do it too quickly :).

I've run into that several times, but only when TortoiseGit, or more precisely its cache, is running or was running very recently. I wouldn't be surprised if similar cache programs exhibit the same behavior on any operating system that prevents deleting a directory tree while another program has any of the directories still open. --Beman

On 14 Mar 2015 at 13:11, Beman Dawes wrote:

On Fri, Mar 13, 2015 at 8:26 PM, Niall Douglas wrote: ... deleting a directory tree

...

which will randomly fail on Windows if you do it too quickly :).

I've run into that several times, but only when TortoiseGit, or more precisely its cache, is running or was running very recently. I wouldn't be surprised if similar cache programs exhibit the same behavior on any operating system that prevents deleting a directory tree while another program has any of the directories still open.

The cause of this is actually very interesting, or maybe it is just to people interested in filing systems like you and me. Anyway, time to bore the list ... NT inherited from VMS the notion of pending deletion never actual deletion, so when you delete a file you actually don't delete it, you merely tag it as likely to be deleted at some future point. Here's how DeleteFile/RemoveDirectory works internally: 1. Open the file/directory as a HANDLE with DELETE privs. 2. Tell the kernel to toggle the PendingDelete boolean using NtSetFileInformation. 3. Close the handle. The PendingDelete flag being set has two consequences. Firstly, all new file handle opens with read or write privs will now fail with STATUS_DELETE_PENDING (ACCESS_DENIED in Win32), though you can still open a new handle if you ask for no privs. Existing handles are unaffected. Secondly, as the reference count for the handle decreases, when it hits zero the PendingDelete flag means to do the following: 1. Mark as hidden the file name in its directory. It will no longer appear in directory enumerations, but attempting to create a file with the same name will return a STATUS_DELETE_PENDING error with no apparent cause. 2. Secure erasing now occurs, which on CIA/NSA editions of Windows means multiple scrubs of the file contents, for each of the named streams attached to the file entry. 3. Actual deallocation of the inode and extents containing data now occurs. These get scheduled to be flushed to the disc as soon as possible. 4. Once the extents deallocation hits the journal, only now does the file name become actually deleted from the directory and a new file with the same name can be created. On marking the entry as empty, Windows again flushes the directory to physical storage (i.e. a fsync). Note that on a busy hard drive it can take milliseconds for the extents deallocation to reach physical storage. Note that so long as the file name remains in the directory, even if hidden, you *cannot* delete that directory because the directory is not empty, even if it is indistinguishable from being empty. You can at this stage see how many ways a directory tree delete can fail. Firstly, any program holding open a handle to any file or directory in the tree will prevent deletion occuring, and therefore directories are not empty, and therefore cannot be removed. As you mentioned, TortoiseGit is a devil for that, but so are virus checkers or anything else which opens file handles. Secondly, if you try to delete a directory tree too quickly - which AFIO usually does because it will parallelise deletes on all CPU cores - you get caught by files taking up to a millisecond in the "file entry hidden but not actually deleted" stage which stops the directory being deleted. Not being able to delete a directory means everything higher up the tree can't also be deleted. It's a big pain. There is also a big problem with these semantics and lock files. If many threads are creating and deleting a lock file quickly, much of the time you get back access denied errors due to the zombie "being deleted" stage rather than more obvious errors like "file already exists". You also get enormously lowered lock file performance such that Windows looks very slow compared to Linux. All of the above plague AFIO's unit testing on Windows because code which works perfectly on POSIX will fail in all sorts of random ways on Windows, and AFIO does a lot of heavy filing system stress testing. So, in the v1.3 release (any day now, everything is finished except for fixing the last of the filing system races as AFIO now has a "race free" mode) I've added the following workarounds: 1. When tagging deletion, first rename it to a 128 bit hex crypto random name. DELETE privs also allows renaming, so this allows a new file with the same name to be immediately created. Performance with this workaround alone is about 20x faster, and suddenly NTFS looks competitive to ext4. 2. I'm shortly about to improve workaround 1 by renaming the about to be deleted item to live somewhere else on the same volume not in its original directory, and still with its crypto random name. I have yet to write logic to figure out some suitable other location on the same mounted volume, but it should be easy enough. This improved workaround stops pending file deletion getting in the way of deleting directory trees, and should make Windows filing system semantics identical to POSIX [1][2]. [1]: Well, apart from renaming. Windows does not permit renaming a directory containing an open file handle, so some future AFIO version may depth rename all the contents of a directory to the temp location, rename the directory, and then rename all the contents back in again. Yes this is completely daft. The only good news is that renames when using the NT kernel API are amazingly quick, and atomic, because metadata flushes of the containing directories don't appear to occur until the handle is closed. As the NT kernel API requires renames to open a handle to the item, you simply hold open the item during the switch out/switch in. [2]: The only other semantic difference is in symbolic link traversal. Windows doesn't have the same semantics as POSIX period. I can't do much about that. Most of the time no one will notice however. Okay, boring the list is over! Hopefully something in the above might be useful in helping Boost.Filesystem work around NT idiosyncracies. As much as they appear to be a pain, there is a logic to them, and judicious use of renaming can allow a reasonable emulation of POSIX semantics. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

On Fri, Mar 13, 2015 at 4:29 PM, Andrey Semashev wrote:

On Friday 13 March 2015 16:17:05 Tom Kent wrote:

...

All my windows test runners for develop (teeks99-08*) have been failing for the last few days (I just noticed).

This may be caused by excessively long paths. I believe, recent changes to the build system added address-model-64\architecture-x86 which were not typically present before.

Which makes me wonder. (a) When will MS get around to fix this in Windows? (I suspect "never" is the answer). (b) AFAIK, there is Windows API that can work with long paths (32767 chars or something?). Can bjam be updated to use these API to work around the problem?

This is indeed the case, can we roll-back last week's changes that caused this until we can implement one of the ideas for making the build system have shorter paths? Develop hasn't had any successful visual studio tests for a while now. Tom

On Tue, Mar 17, 2015 at 7:58 PM, Steven Watanabe wrote:

AMDG

On 03/17/2015 06:35 PM, Tom Kent wrote:

...

This is indeed the case, can we roll-back last week's changes that caused this until we can implement one of the ideas for making the build system have shorter paths?

It already exists, and it's even documented:

http://www.boost.org/build/doc/html/bbv2/overview/invocation.html#bbv2.overv...

The really key question though is.. Will --hash break the results parsing and generation or not? Which unfortunately I don't know the answer to since I didn't write the code that handles the bjam test log processing. But if the answer is yes it will work.. I'd gladly make the 5 minute change, of adding --hash to all regression testers, ASAP. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

On Tue, Mar 17, 2015 at 7:58 PM, Steven Watanabe wrote:

AMDG

On 03/17/2015 06:35 PM, Tom Kent wrote:

...

This is indeed the case, can we roll-back last week's changes that caused this until we can implement one of the ideas for making the build system have shorter paths?

It already exists, and it's even documented:

http://www.boost.org/build/doc/html/bbv2/overview/invocation.html#bbv2.overv...

That seems to have worked, thanks. Tom