On Thu, Aug 26, 2004 at 12:18:12PM -0400, Beman Dawes wrote:

I remember that you had some feeling that the generic path grammar could be improved to make relative/absolute queries more uniform between Windows and POSIX. My plan was to sit down with you in Sydney and learn more about your ideas, but then I ended up not going. Maybe we could work on that by email.

What do you think about my proposal(s)? I (too?) think that an important improvement can be made by making an explicit distinguishment between relative and absolute paths. My feeling is that only relative paths are truely portable and therefore, the developer should be "forced" more or less to set up the absolute path "nodes" that he wants to work with as a sort of initialization. Apart from that there is then user input of course, which might happen at any point in the code - but that is then either a change of such a "node", or it is an explicit filepath which is to be treated at native the whole time. My idea is therefore that there should be two types: - An absolute/native path type and - A relative/portable path type It should not be possible to convert the native_path class into the relative_path class type (these are just my naming proposals), but the other way around can be seemless supported by allowing to specify a "node" (of the absolute/native path type) at the moment a relative path is created. The developer then can concentrate and work on with just the portable, relative, value of the path and does not have to worry about the native conversion when this is necessary prior to calling a system call (like say fs::exists), because this could be done automatically then. Actually, this idea is very closely related to the whole windows concept of multiple roots: C:, D:, E: etc are not only "just" unmounted partitions (that is, on linux they would be just mounted somewhere and therefore, if C: is the root say, a D: would be /usr etc) but they have a _variable_ value each: the working directory. If one specifies A:foo or B:foo, then this refers to whatever the value of the "node" (working directory) of 'A' respectively 'B' is at that moment - with the restriction that they have to be on that particular partition. Of course we'd not have that restriction. Example: fs::native_path A("C:/mydir"); fs::native_path B("B:/"); fs::native path C("C:/actually this can be anything"); fs::relative_path bar("bar.txt"); // Use A / bar, or B / bar or C / bar. However, I think it makes more sense to do: fs::relative_path bar(C); // Relative to C. bar = "bar.txt"; // Just use bar. if (fs::exists(bar)) ... What do you think? -- Carlo Wood <carlo@alinoe.com>

On Mon, Aug 30, 2004 at 12:12:58PM -0400, Beman Dawes wrote:

...

My idea is therefore that there should be two types: - An absolute/native path type and - A relative/portable path type

We looked at that approach during initial design, and it just didn't work. The added complexity was really messy, yet the benefits were few.

For example, what does the signature of the exists() function look like if there are two path types? Do we now need two exists() functions? What about functions that take two path arguments? Do those now need four versions? That might just barely be acceptable in the filesystem library itself, but you couldn't inflict that sort of argument explosion on user code. Instead, you would need a base class. That adds complexity. It just gets too messy, and tests to break use cases we really want to work. We really want the user to be able to write:

if fs::exists("foo") ...

rather than:

if fs::exists(fs::relative_path("foo")) ...

--Beman

My idea on this was (and still is) that it is possible to convert a relative path to an absolute path (automatically) by adding an absolute path object reference to the relative path object. This should make sense in every application imho. As a result, you can simply use absolute path types for all those functions that will pass that parameter to a system call. In the case of fs::exists, you would have only fs::exists(fs::native_path const&) and when calling that with a fs::relative_path - it would get expanded to the absolute path even before entering fs::exists (this expanding is necessary anyway - so why not before you do the actual call?). That way there is always only one function needed. Calling fs::exists("foo") would cause the constructor fs::native_path(char const*) to be called, in which case fs::exists would do the expansion internally after seeing that it is not a complete path, using the fs::initial_directory() (or whatever it was called.. the pwd). Same thing for relative_path objects not associated with an absolute fs::native_path object already. Recall that my proposal was that you can do either fs::native_path np("foo%$#@!"); // A relative path using weird chars. if (exists(np)) // Will expand `np' by prepending fs::initial_directory(). or fs::native_path absolute_node; fs::relative_path rp(absolute_node); // Tie `rp' to `absolute_node'. absolute_node = "C:\Program Files"; if (exists(rp)) // Throws if absolute_node is not complete. for the same reason, this should _throw_ and not, like it currently does - just return false because the path doesn't exist: fs::native_path np("/usr/src"); // Note, not possible to use fs::relative_path here (would throw). if (exists(np)) // Should throw on cygwin (with BOOST_WINDOWS) // because the path is not complete! Correct would be for example: #ifdef BOOST_WINDOWS fs::native_path root("C:/cygwin"); #else // ... #endif fs::native_path np = root / "usr/src"; if (exists(np)) // Ok -- Carlo Wood <carlo@alinoe.com>

On Mon, Aug 30, 2004 at 07:17:49PM -0400, David Abrahams wrote:

It seems to me that some APIs might very well want to use an interface that can be statically checked for absolute paths, because relative paths can be so vague (and not safe from race conditions). Carlo's idea appeals to me for that reason as well.

Well thanks... but I didn't want to disturb the current implementation of boost::filesystem that much - and so far I assumed that my fs::native_path CAN contain a relative path. My most important analysis of practical use of paths from a portable point of view was that an absolute path should be considered to be _always_ native - hard coded in a type. But not necessarily the other way around (a non-absolute path can still be native as well). The naming fs::native_path is essential thus and should not be confused with a fs::absolute_path, that I did not propose. -- Carlo Wood <carlo@alinoe.com>

On Tue, Aug 31, 2004 at 08:40:03AM -0400, David Abrahams wrote:

...

My most important analysis of practical use of paths from a portable point of view was that an absolute path should be considered to be _always_ native - hard coded in a type. But not necessarily the other way around (a non-absolute path can still be native as well).

I know you hate to answer such questions, but what problems does that solve?

Portability robustness of code. I've given numerous of examples in the meantime in other posts.

From my post of 30 Aug 2004 (two posts earlier in this thread, you even replied to it):

Correct would be for example: #ifdef BOOST_WINDOWS fs::native_path root("C:/cygwin"); #else // ... #endif fs::native_path np = root / "usr/src"; if (exists(np)) // Ok and the fact that this should throw: fs::native_path np("/usr/src"); // Note, not possible to use fs::relative_path here (would throw). if (exists(np)) // Should throw on cygwin (with BOOST_WINDOWS) // because the path is not complete! Compare that with the current implementation: #ifdef BOOST_WINDOWS fs::path root("C:/cygwin"); #else // ... #endif fs::path np = root / "usr/src"; if (exists(np)) // Ok Now we 'forget' to use the 'root' there: fs::path np = "/usr/root"; if (exists(np)) // Returns 'false' rather than throwing. The fact that it is possible to make a mistake like that comes from the fact that the current implementation doesn't enforce a truely portable "relative_path" type, but allows one to use any "path value" for fs::path "variables" and (ab)use them anywhere regardless. The only way to catch the above mistake without introducing a new type would be to test inside fs::exists that the path that is being passed is complete - and if not, demand that it is relative. But that would be kludge and not attack the REAL problem at the root.

...

The naming fs::native_path is essential thus and should not be confused with a fs::absolute_path, that I did not propose.

Hm.

Note that the current implementation doesn't use the word 'absolute' - it speaks of 'complete'. I take those two as the same in this discussion. Of course it would be possible to introduce another level of paths and call that 'complete_path' - which would be forced to be complete - or make itself complete at construction if possible. However, I don't see must use for that - because it seems better, more flexible, to delay that completion until the path is actually being passed to a system call. So, your proposal only would make sense to me if you would not allow automatic conversion to a complete path (ie: fs::complete_path("foo") would throw) so that this type would be complete at all times. But well, what does that solve? }:-) -- Carlo Wood <carlo@alinoe.com>

On Wed, Sep 01, 2004 at 02:20:29PM -0400, Beman Dawes wrote:

...

My idea on this was (and still is) that it is possible to convert a relative path to an absolute path (automatically) by adding an absolute path object reference to the relative path object.

Yes, as long as you know the absolute path reference. Is the absolute path

?? You didn't read my posts. I'd *add* that reference to a relative_path. Reference as in '&'. class relative_path { native_path& absolute_path; ...

reference known for all operating systems? I'm not sure about that. For example, there have been multi-rooted systems that didn't have a concept of "current drive". How would that work?

Not relevant thus.

...

This should make sense in every application imho.

It makes sense for calling operating system file system functions. It may make no sense at all if the use of the path is for reporting or other path manipulations.

Then those functions won't complete the path. I always said "when a system call needs to be called". You didn't read what I posted before.

The programs which process the Boost regression test bjam output do a lot of that. Another example is reporting the path name from a filesystem function which throws an exception.

I never said that path::string would complete the path.

...

As a result, you can simply use absolute path types for all those functions that will pass that parameter to a system call.

Don't for get error reporting. While you certainly could report the absolute path rather than the path as given, that would be a step backward IMO.

I never said that path::string would complete the path.

...

In the case of fs::exists, you would have only fs::exists(fs::native_path

...

const&) and when calling that with a fs::relative_path - it would get expanded to the absolute path even before entering fs::exists (this expanding is necessary anyway - so why not before you do the actual call?). That way there is always only one function needed.

You lost me.

Clearly.

I could see how what you are describing would work for exists( fs::absolute_path const & ). It seems you are making some kind of assumption regarding the relationship between native paths and absolute paths. That is possible to do for Windows and POSIX, but not for operating systems in general.

??? You know what assumption I have made? Apparently not, then how can you claim that this assumption is not true for any operating system (it is)?

...

Calling fs::exists("foo") would cause the constructor fs::native_path(char const*) to be called, ...

What happens when the app wants "foo" treated as a generic rather than native format?

Is there a difference in this case? In the end the path must be completed, or you cannot check if it exists, period. I am sorry - but it is clear you have no idea what I am talking about from which I have to conclude that you didn't read my previous posts. Never mind then. -- Carlo Wood <carlo@alinoe.com>

On Wed, Sep 01, 2004 at 10:12:41PM -0400, David Abrahams wrote:

Carlo, you're displaying an unreasonably petulant attitude.

In that alinea, I was.

Part of what makes Boost effective is that people work together to understand problems. Not least because of Beman's recent personal circumstances, he deserves a patient helping hand to understand what you're proposing.

I was not aware of his personal circumstances. I had the idea he has been on a holiday - and that is all I know.

If you're going to say "never mind then" every time someone fails to quickly grasp and accept your ideas, this is not an environment in which you will be able to operate effectively.

Yes, yes... but it is not because I am irritated that I said that. I say it because the communication was already such non-effective (inefficient and fruitless) that it became below a certain threshold; I like to do things that are productive. I can be very patient with say 15 y/o C++ newbies who just started to learn C++ (ie, on IRC) just as long as they clearly make progress, but when a professional C++ developer and maintainer of an important package doesn't read (and/or understands) posts about his very own package then that instantly irritates me - sorry, can't help it :). I appologies for having displayed that though. Anyway, because you keep asking me (more or less), I will reply again to the same post - displaying lots of patience this time :p -- Carlo Wood <carlo@alinoe.com>

From: Carlo Wood <carlo@alinoe.com>

On Wed, Sep 01, 2004 at 10:12:41PM -0400, David Abrahams wrote:

...

If you're going to say "never mind then" every time someone fails to quickly grasp and accept your ideas, this is not an environment in which you will be able to operate effectively.

Yes, yes... but it is not because I am irritated that I said that. I say it because the communication was already such non-effective (inefficient and fruitless) that it became below a certain threshold; I like to do things that are productive. I can be very patient with say 15 y/o C++ newbies who just started to learn C++ (ie, on IRC) just as long as they clearly make progress, but when a professional C++ developer and maintainer of an important package doesn't read (and/or understands) posts about his very own package then that instantly irritates me - sorry, can't help it :). I appologies for having displayed that though.

Communication is, by definition, a closed loop system. When you write something, you hope that the words you write adequately capture the ideas in your head and that your reader(s) will infer from those words your ideas. If that breaks down, information doesn't flow as intended. So, the recipient counters, based upon what may be a flawed understanding of what you wrote and meant. That effort may be faulty, too, but assuming you understand what the recipient wrote in response, you can clarify what you wrote. Eventually, one hopes that the same ideas are shared. You seem to have made the assumption that everyone reads with rapt attention and total understanding what you've written, when that is often not the case. This is a busy list read by busy people, so folks often skim -- to the deteriment of communication at times, to be sure -- and miss important details. I'm sure it gets frustrating having to repeat yourself or find new ways to express an idea, but that's the price one pays when being human and dealing with other humans. I hope you recognize that we're all busy, and yet that we are interested in improving ourselves and the code in Boost. No slight was intended against you; reality just intervened. Please continue to do your best to be patient and look for areas of confusion and ways to eliminate it. -- Rob Stewart stewart@sig.com Software Engineer http://www.sig.com Susquehanna International Group, LLP using std::disclaimer;

On Wed, Sep 01, 2004 at 02:20:29PM -0400, Beman Dawes wrote:

At 01:53 PM 8/30/2004, Carlo Wood wrote:

...

My idea on this was (and still is) that it is possible to convert a relative path to an absolute path (automatically) by adding an absolute path object reference to the relative path object.

Yes, as long as you know the absolute path reference. Is the absolute path reference known for all operating systems? I'm not sure about that. For example, there have been multi-rooted systems that didn't have a concept of "current drive". How would that work?

What I mean is that in practical cases a relative path will relative to some directory known to the developer, most likely one that he even reserved a variable for. In other words, in *most* cases there is a relationship between 'relative paths' and 'absolute paths' such that there belongs exactly one 'absolute path' to each 'relative path' (not the other way around, one 'absolute path' can be used by many 'relative paths'). Because this will be the case in over 99% or so cases, it makes sense to add special support for it by adding an 'absolute path' member to the 'relative path' class. This member then can be initialized to refer to an 'absolute path' object (defined by the user). For example: fs::absolute_path ap; fs::relative_path rp(ap); // Tie to 'ap'. ap = "/c/cygwin"; // Set or change it. rp = "foo"; // Now indirectly refers to ap / "foo". // The value of rp is still only "foo" though. In the case that the developer does (want to) specify an 'absolute path' that the 'relative path' will be relative to, for example: fs::relative_path rp; // Not tied to anything. Then it indirectly it will be tied to whatever the native system understands to be 'the current directory', if any. And a call to fs::exists(rp) should do the same as it does with the current implementation. PS In the above I used 'absolute_path' for clarity reasona, but in my real proposal that type does not occur. It should be replaced with 'fs::native_path' (but still be assigned an absolute value) with the same consequences.

...

This should make sense in every application imho.

It makes sense for calling operating system file system functions. It may make no sense at all if the use of the path is for reporting or other path manipulations.

My proposal was, and is, that this extra reference (added to relative_path, see above) will ONLY be used when a relative path needs to be completed because it is passed to a (file) system call. This happens internally in boost::filesystem and, like you said, makes sense. In cases it does not makes sense - it will simply not be done. For example in the case of reporting (ie, calling the 'native_file_string()' method) the same string will be returned as in the current application. It might be useful however to introduce a method complete_native_file_string() that would complete the path before returning the string, in the same way that a fs::exists would complete the path before testing it.

The programs which process the Boost regression test bjam output do a lot of that. Another example is reporting the path name from a filesystem function which throws an exception.

It might actually make sense to print the full path that the system tried to access in that case - but that is debatable.

...

As a result, you can simply use absolute path types for all those functions that will pass that parameter to a system call.

Don't for get error reporting. While you certainly could report the absolute path rather than the path as given, that would be a step backward IMO.

...

In the case of fs::exists, you would have only fs::exists(fs::native_path const&) and when calling that with a fs::relative_path - it would get expanded to the absolute path even before entering fs::exists (this expanding is necessary anyway - so why not before you do the actual call?). That way there is always only one function needed.

You lost me. I could see how what you are describing would work for exists( fs::absolute_path const &).

There is no 'fs::absolute_path' in my proposal. The only relationship that I made between (the proposed) fs::native_path and a path being absolute is that any absolute path should be forced to be of the type 'fs::native_path', because essential any absolute path is native in nature. The objection you came with was that by introducing another type (where I see 'fs::native_path' as the addition and 'fs::relative_path' as the most equivalent to the current 'fs::path') that then function prototypes would be exploding in number. For example both fs::exists(fs::native_path const&) and fs::exists(fs::relative_path const&) would be needed according to you. The above paragraph states that the latter is not needed. You can do with _only_ fs::exists(fs::native_path const&). Why? Because the other one would resolve itself; for example, take this code: fs::native_path ap("/c/cygwin"); fs::relative_path rp(ap); if (fs::exists(rp)) ... You'd think that for this case the fs::exists(fs::relative_path const&) is needed but that is not true because any 'relative' path is convertable to 'absolute path'. Implicit conversion would take place and a temporary fs::native_path with value 'ap / rp' would be passed to fs::exists. Hence my claim that the exploding of number of function will not happen.

It seems you are making some kind of assumption regarding the relationship between native paths and absolute paths. That is possible to do for Windows and POSIX, but not for operating systems in general.

I don't think that I am making assumptions. I merely made an observation, an analysis of 'file paths' in general, and came to the conclusion that a relative path in over 99% of practical cases only make sense when it is relative to *something*. Still - the other 1% (or less) is not harmed. You can still just use only fs::relative_path, without the need to tie it to some absolute path, and still just print its value.

...

Calling fs::exists("foo") would cause the constructor fs::native_path(char const*) to be called, ...

What happens when the app wants "foo" treated as a generic rather than native format?

In the case of "foo" that makes no difference. The reason that fs::native_path(char const*) is called is just because there is no fs::exists(char const*) and also no fs::exists(fs::relative_path const&) The effect however would be the same when those did exist - so there is no need to add them.

A better example might be "foo.bar" on an operating system which uses "." as a directory separator in native formats.

If a string-literal is passed that has a different meaning when treated as a native path, then when it is treated as a generic path, then adding an extra signature for fs::exists won't help, will it? You can't fix that ambiguity without explicitely saying what type you mean. However, I think that the confusion comes from the name 'native_path'... It is NOT my proposal to have the constructors of this type behave any different from what the current implementation is doing (expect when it comes to throwing an exception). In the current implementation, on a system where "." is the native path separator, you would do: fs::path p("foo/bar"); cout << p.native_file_string(); and that would print "foo.bar", right? Then the same is the case when using fs::native_path: fs::native_path p("foo/bar"); cout << p.native_file_string(); Problems would occur when '.' is the path separator and '/' can be used as file characters in both senarios no? You'd also still be able to do fs::native_path p("c:\\Program Files", &fs::native_check); to *actually* get a native check. I was not aware of OS where '.' is path separator, but I suppose that this works? fs::native_path p("foo.bar", &fs::native_check); Then, in order to get that behaviour, one would have to explicitely specify it: if (fs::exists(ap / fs::native_path("foo.bar", &fs::native_check))) I don't think that is different for the current implementation with just fs::path. -- Carlo Wood <carlo@alinoe.com>

Beman Dawes <bdawes@acm.org> writes:

At 02:48 PM 8/26/2004, Carlo Wood wrote:

...

... What do you think about my proposal(s)?

I (too?) think that an important improvement can be made by making an explicit distinguishment between relative and absolute paths.

Dave was talking about changes to the path grammar. IIRC, he was not suggesting adding additional path types.

I do think Carlo's idea is intriguing. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com

Beman Dawes <bdawes@acm.org> writes:

At 08:21 PM 8/24/2004, David Abrahams wrote:

...

...I have a hard time with some design elements of boost::filesystem, but have a hard time formulating cogent arguments about them.

I remember that you had some feeling that the generic path grammar could be improved to make relative/absolute queries more uniform between Windows and POSIX.

I don't remember that; I guess it's been too long.

My plan was to sit down with you in Sydney and learn more about your ideas, but then I ended up not going. Maybe we could work on that by email.

Maybe next week; I'm all tied up now. Cheers, -- Dave Abrahams Boost Consulting http://www.boost-consulting.com