On 14/11/2013 06:31, Quoth Niall Douglas:

I would report this as a bug to 7zip. The MAX_PATH limit is purely for backwards compatibility only, and every Win32 API can easily take 32k paths by prepending a magic marker.

There are some downsides to this, such as losing current directories and relative paths. (Also, there are many applications that still only use the ANSI APIs, which do not accept the magic marker.) Having said that, 7zip is the sort of program that shouldn't have trouble with any of that. (But it still might make building or browsing the documentation harder later on.)

On 14/11/2013 13:45, Quoth Simon Sasburg:

Even worse, windows explorer still seems to be using the ANSI api's (or it is itself introducing path limits?), and cannot reliably interact with files in long paths.

It's definitely not using the ANSI APIs but it's not using the extended syntax for the most part either. (And if you look at the docs for the shell library interfaces, most of the ones that accept complete pathnames are still limited to MAX_PATH. Although there are workarounds for that such as using item ids instead of filenames.)

On 14/11/2013 14:31, Quoth Ben Craig:

I think this is being unrealistic. If we take this approach, then you should request every Win32 program that uses fopen, iostreams, or CreateFile without the magic marker to change their program to use a non-portable API. This includes programs like cmd.exe, explorer, bjam, 7zip, and countless others.

Did you mean to quote someone else? Because I was agreeing with you.

I wish that "normal" Windows APIs permitted a higher maximum path length, but they don't. Reasonable Windows support means acknowledging this limitation.

It's unfortunately a wart of its DOS outgrowth. Back in the days of 8.3 filename limits a total path length of 260 was ample, and most of the early DOS APIs were based around local stack buffers of [MAX_PATH]. This has ended up getting carried forward via backwards-compatibility, because the APIs don't provide any way to accept the buffer size explicitly and it can't be increased arbitrarily without crashing older programs.

On 13 Nov 2013 at 19:31, Ben Craig wrote:

I think this is being unrealistic. If we take this approach, then you should request every Win32 program that uses fopen, iostreams, or CreateFile without the magic marker to change their program to use a non-portable API. This includes programs like cmd.exe, explorer, bjam, 7zip, and countless others.

Mmm, you make a good point. Microsoft ought to flip the default instead of everybody else making the change.

I wish that "normal" Windows APIs permitted a higher maximum path length, but they don't. Reasonable Windows support means acknowledging this limitation.

I've bumped the notion of raising that limit to our friends in Microsoft. Who knows, it may be possible for Windows 8.2! Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

On 15 Nov 2013 at 13:52, Gavin Lambert wrote:

On 15/11/2013 08:52, Quoth Niall Douglas:

...

I've bumped the notion of raising that limit to our friends in Microsoft. Who knows, it may be possible for Windows 8.2!

As I previously mentioned, I don't think it's something they can do on a version boundary, only on an ABI boundary (eg. 32-bit vs. 64-bit, or by introducing new APIs).

There's just too much existing (and will not be recompiled) code that has stack based char[MAX_PATH] buffers.

They can't even do the manifest-version-based shimming that they're doing for some other things, as that's defined at the process level -- just because an application claims that it knows about a larger MAX_PATH doesn't mean that all the DLLs that it uses does.

I wouldn't be proposing that MAX_PATH should be changed for any code not being compiled for the latest toolset where MAX_PATH was extended. So, if you're linking to MSVCRT13.DLL then you get MAX_PATH=1024, while any previous MSVCRT gets MAX_PATH=260. What about mixed MSVCRT apps? Well, already those aren't reliable and you shouldn't do them :) But it still should usually work as usually application code is compiled with a newer toolset than library code, and therefore you're passing in a larger buffer which may cause library code to puke on the longer path, but not corrupt memory. A lot of time library code simply passes through externally supplied buffers anyway, and therefore doesn't care.

(Not that I wouldn't love them to try, I just wouldn't hold out much hope for it.)

I got a response - it is thought highly unlikely to be successful. I guess you'd need to make a convincing business case, and I agree given Windows' size in the market they don't have to bother adapting to others i.e. if you're porting to Windows, you accept the 260 char path limit as part of the porting effort and all the code refactoring therefore needed. BTW, my current side project of implementing a standard transactional graph database gets rid of the path length limitation on Windows. One of its interfaces is a POSIX-like file system interface, and by POSIX-like I mean it looks just like a POSIX file system except it has more reserved characters for working with things like multiple file versions, creating directories which are really live searches on the graph store, being able to transact update many files atomically or not at all etc. My idea is that (eventually) you could mount this graph store as a fuse filesystem and work with it as if it were a real file system. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

On 18 Nov 2013 at 11:25, Gavin Lambert wrote:

...

I wouldn't be proposing that MAX_PATH should be changed for any code not being compiled for the latest toolset where MAX_PATH was extended. So, if you're linking to MSVCRT13.DLL then you get MAX_PATH=1024, while any previous MSVCRT gets MAX_PATH=260.

That's not where MAX_PATH comes from though. It's in the Windows SDK, not the C Runtime. There's plenty of code that calls the WinAPI directly, and that's where the value of MAX_PATH matters.

You are right, though by MSVCRT version I was meaning MSVC toolset version e.g. the v120 thing you can choose. And of course well written Win32 API code ought to adjust to a larger MAX_PATH without any problems at all :) Still, it would be hard to create a business case in favour of it within Microsoft, and therefore it is very hard to imagine it happening.

...

BTW, my current side project of implementing a standard transactional graph database gets rid of the path length limitation on Windows. One of its interfaces is a POSIX-like file system interface, and by POSIX-like I mean it looks just like a POSIX file system except it has more reserved characters for working with things like multiple file versions, creating directories which are really live searches on the graph store, being able to transact update many files atomically or not at all etc. My idea is that (eventually) you could mount this graph store as a fuse filesystem and work with it as if it were a real file system.

Sounds a little like the database-based filesystem MS were planning to replace everything with in Vista, but eventually abandoned. (WinFS, I think it might have been called..?)

I could talk about that one for several hours ... yes, you're right I've spoken extensively with colleagues who used to work on WinFS back in the day. They believe my design approach to be correct. Whether it *is* correct of course I may find out some day depending on what happens after we relocate from Canada back to Europe. Niall -- Currently unemployed and looking for work. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/