Tobias Schwinger wrote:

David Abrahams wrote:

...

It's intentional. When your iterator's reference type is not value_type& or value_type const&, your reference type is the same as the value_type or you are using a proxy reference. Either way, if your iterator is readable, the reference is implicitly convertible to value_type. If your iterator is writable, it can be assigned from the value_type. No persistent object (lvalue) of type value_type is revealed by dereferencing the iterator; you can at best see a temporary. If iterator_facde handed you a pointer to non-const value_type, it would allow you to modify this temporary, which would then disappear. We don't do that for the same reasons the language won't bind a non-const reference to a temporary.

I see - thanks for your detailed reply.

When using proxy classes as references the assumption that modifications of a temporary object are lost at the end of this very object's lifetime does not apply

It most certainly does apply. There may be some cases in which it doesn't, but the usual case is that it does.

- modification of the proxy object actually means changing the stored data internally referenced by it.

Normally the reason for a proxy is that there isn't any "stored data internally referenced by it." If there's a persistent lvalue, why not just return a real reference instead of a proxy?

In this case it would be OK if not preferable to have operator-> return a pointer to non-const...

Is there any way to achieve this (besides hand-coding the whole iterator) ?

Just hand-code your own operator->.

Or any hope that there will be a parameter to explicitly disable this extra-safety ?

And why does operator* give me a mutable (temporary) object then, by the way ?

It is assumed that if you passed non-const R for the reference type, you are trying to make a writable iterator, and R will have a (usually non-const) assignment operator that takes a value_type argument. If we returned a const object, it would be impossible to write *p = x; -- Dave Abrahams Boost Consulting http://www.boost-consulting.com

Tobias Schwinger wrote:

David Abrahams wrote:

...

Tobias Schwinger wrote:

...

I see - thanks for your detailed reply.

When using proxy classes as references the assumption that modifications of a temporary object are lost at the end of this very object's lifetime does not apply

It most certainly does apply. There may be some cases in which it doesn't, but the usual case is that it does.

Writing "proxy class with reference semantics" would have been more precise, I figure.

I guess not, really. "Reference semantics" is pretty loosely-defined, and it's sort of the job of any proxy reference to have some kind of "reference semantics." Could you explain some more?

...

...

- modification of the proxy object actually means changing the stored data internally referenced by it.

Normally the reason for a proxy is that there isn't any "stored data internally referenced by it." If there's a persistent lvalue, why not just return a real reference instead of a proxy?

Because I have to change its interface. In my particular case the persistent lvalue is of array type.

But why not just change the interface in your dereference function and hand back a real reference at that point? If you are not storing a copy of the value_type within the proxy, I don't see what the proxy can be buying you.

...

Just hand-code your own operator->.

OK - that's what I did, for now.

...

...

And why does operator* give me a mutable (temporary) object then, by the way ?

It is assumed that if you passed non-const R for the reference type, you are trying to make a writable iterator, and R will have a (usually non-const) assignment operator that takes a value_type argument. If we returned a const object, it would be impossible to write

*p = x;

If the expression

*p

gives me a temporary object with no reference (write-through) semantics it does not make sense to write to this temporary by writing

*p = x;

according to your above rationale, does it ?

That is true. But when the reference type is not the same as the value_type, an iterator author has to go out of his way to make that assignment compile.

Same goes for:

(*p).mutator(value);

And *way* out of his way to make this one compile, since the reference type must now have forwarding members for each member of the value_type.

and

p->mutator(value);

So now we come to this one. This one is entirely in the hands of the library to prevent, since the user doesn't usually determine the return type of operator->. It seems like a bad idea to allow it silently. The library is supplying an operator-> that stores an object of the value_type. There are very few situations where modifications to that object can go anywhere useful, so it makes sense to prevent them. The library is able to make sensible default choices without fear because the user can always reimplement any operator within his derived iterator.

...and I can't imagine there is a user who would seriously complain that

pointer ptr ( i.operator->().operator->() );

gives an invalid pointer ;-).

Am I missing something ?

There was a guy who complained just last week on this very mailing list that putting an iterator_adaptor over a T const* suddenly produced writability in the iterator through operator-> !! http://lists.boost.org/MailArchives/boost-users/msg08656.php -- Dave Abrahams Boost Consulting http://www.boost-consulting.com

Tobias Schwinger wrote:

David Abrahams wrote:

...

But why not just change the interface in your dereference function and hand back a real reference at that point? If you are not storing a copy of the value_type within the proxy, I don't see what the proxy can be buying you.

I'll put a short problem-domain-specific excursion at the end of this post which explains what exactly the proxy is buying me.

Thanks.

...

So now we come to this one. This one is entirely in the hands of the library to prevent, since the user doesn't usually determine the return type of operator->. It seems like a bad idea to allow it silently. The library is supplying an operator-> that stores an object of the value_type. There are very few situations where modifications to that object can go anywhere useful, so it makes sense to prevent them.

Doesn't the fact that reference and value_type are the same and both are mutable class types indicate that this should work as well ?

Ah. Well now, that is a special case. Maybe we should allow it whenever reference is the same as or derived from value_type. Incidentally, I don't call that case a proxy -- as far as I'm concerned that's just an ordinary non-lvalue iterator. I suppose if your value_type is really a refcounted pimpl it is somewhat proxy-like, though. Incidentally #2, the library is focused mainly on building generic iterators so we usually didn't consider the fact that class value_type objects returned by value from an iterator's operator* might be assigned into.

After all, it should be just another notation for

(*i).mutator(value)

from a user's perspective.

...

The library is able to make sensible default choices without fear because the user can always reimplement any operator within his derived iterator.

My workaround to allow it feels somewhat ugly, because I have to redundantly do the same thing iterator_facade does already (use yet another proxy to aggregate my temporary reference object and make its operator-> return a pointer to it - just like operator_arrow_proxy (given that reference and value_type are the same), except that there is a non-const operator->, too).

Understandable.

!!! However, it works for me. I don't want to try to make you change things for what could be a special case !!!

I'm not too sure calling mutators on a proxy class really is that special, though.

...

...

...and I can't imagine there is a user who would seriously complain

It's not so special, but building a proxy reference that fully emulates the interface of the referenced value type is quite unusual. It's takes a lot of work, is very fragile and can't be done generically. that

...

...

pointer ptr ( i.operator->().operator->() );

gives an invalid pointer ;-).

Am I missing something ?

There was a guy who complained just last week on this very mailing list that putting an iterator_adaptor over a T const* suddenly produced writability in the iterator through operator-> !! http://lists.boost.org/MailArchives/boost-users/msg08656.php

Yeah, I (partially) read it to ensure I am not asking redundant questions before starting this thread. I still believe adapting pointers is a different story.

It's not such a different story. If I take a non-writable version your iterator, operating correctly, and wrap iterator_adaptor around it, it would be very strange if operator-> allowed you to mutate members.

Excursion: what's the proxy buying me ?

I use a generic, resizable container (similar to std::vector but with memory layout guarantees) to feed vector data to my graphics board through some rendering API. The proxy allows me to see a vector object instead of a reference to an array of floating type.

Operating on plain arrays in order to perform vector arithmetics is quite unhandy. Using a converting constructor from a reference to array type is a radical and error prone approach and would restrict me to use only one class for vector arithmetics (there is no way to tell if float[3] should be a homogenous 2D-vector or a non-homogenous 3D-vector, for example).

That's why I use a vector-reference proxy class that allows me to work on vectors whose data lives somewhere else.

I understand, thanks. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com

Ronald Garcia wrote:

Howdy,

Dave sent me a forward to this conversation some time ago and I have finally made the time to catch up on the conversation and contribute my two cents. The issues that Tobias describes regarding operator-> are, I believe, exactly the same issues I encountered while implementing Boost.MultiArray.

To use an earlier version of iterator adaptors, I had to supply a modified copy of the library as part of MultiArray's implementation. Using the more recent version of iterator adaptors, I simply had to implement my own operator->() member function and operator_arrow_proxy object. Clearly I can implement the behavior I want using the library as-is.

The question of interest then seems to be the following: Should iterator adaptors somehow more explicitly take my and Tobias' implementation needs into consideration? Perhaps the library's default behavior when Reference is not a true reference type should be changed; Perhaps a FAQ entry in the documentation would suffice. Since I can make my library work correctly today with little pain or code duplication, I'm not particularly passionate about this case.

With respect to the greater language issue, let me respond to an earlier statement by Dave:

"If iterator_facade handed you a pointer to non-const value_type, it would allow you to modify this temporary, which would then disappear. We don't do that for the same reasons the language won't bind a non-const reference to a temporary."

Around here (the Open Systems Lab at Indiana University), I've been arguing against this property of C++ for a long time, specifically that the language does not allow you to bind temporary to a non-const reference. I am under the impression that this behavior is meant to "protect" the programmer from making certain mistakes, but the grave downside to this is that it also prevents a library writer from using proxy objects to mimic reference semantics.

The only case I can think of where that's true is when the reference type is the same as the value type. If we allowed operator->

Taking an example from MultiArray, if I have an 2x2 multidimensional array: boost::multi_array A(extents[2][2]);

and a template function that takes some Random Access Sequence:

template <Sequence> process_sequence(Sequence& Seq) { /* ... */ }

then I cannot pass this function the second "row" of my array using the syntax: process_sequence(A[1]);

Why can't you just return a const proxy?

because the return type of multi_array::operator[]() is an object of type multi_array::subarray. The whole purpose of the subarray is to behave as though it were a nested array within the multi_array. Bear in mind that multi_arrays are not implemented using nested arrays: the internal data is stored as a contiguous heap-allocated array. You could argue that the entire point of the MultiArray library is to allow one to avoid implementing multi-dimensional arrays using such a nested form (i.e. std::vector).

In order to make MultiArray work, it must return a proxy object that implements const and nonconst versions of the array operators.

class multi_array { const mutable_proxy operator[](int); const immutable_proxy operator[](int); ... }; Now mutable_proxy only needs one version of the array operators. Couldn't that work?

The subarray operators manipulate the data stored in the original multi_array that created the original subarray.

In fact, the Matrix Template Library (MTL) ran into the same problem. The library often creates a "view" of a matrix as an argument to a subsequent function. But again, if the subsequent function takes its argument by reference (which it should if the argument is used as an "out parameter"), then the result is a compile-time error. What was their solution? I hate to admit this: const_cast.

Did someone think about returning const proxies?

Let's take a step back for a moment. Why is it okay in the language for me to create a temporary object and immediately call a non-const member function on it, as in: A().member_function();

But it's not okay for me to pass a temporary object to another function that in turn calls that same member function, as in:

template <class T> call(T& x) { x.member_function(); }

call(A());

You may note that I'm an author of a major proposal to allow templated references to bind to non-const rvalues.

I've been convinced by others that if the move semantics proposal is accepted into the language, it will take care of this issue.

Bingo.

Perhaps iterator_adaptors should mimic the behavior of C++ as it stands for the purpose of consistency, but I'm not convinced that this property of the language does much for safety. It is clear to me, however, that it hinders library developers who hope to mimic reference semantics with proxies.

I'm not sure; have they used enough imagination? -- Dave Abrahams Boost Consulting http://www.boost-consulting.com

Ronald Garcia wrote:

Howdy again,

On Dec 13, 2004, at 10:49 PM, David Abrahams wrote:

...

Ronald Garcia wrote:

...

... Around here (the Open Systems Lab at Indiana University), I've been arguing against this property of C++ for a long time, specifically that the language does not allow you to bind temporary to a non-const reference. I am under the impression that this behavior is meant to "protect" the programmer from making certain mistakes, but the grave downside to this is that it also prevents a library writer from using proxy objects to mimic reference semantics.

The only case I can think of where that's true is when the reference type is the same as the value type. If we allowed operator->

Is this a complete thought, or am I missing something?

...

...

Taking an example from MultiArray, if I have an 2x2 multidimensional array: boost::multi_array A(extents[2][2]);

and a template function that takes some Random Access Sequence:

template <Sequence> process_sequence(Sequence& Seq) { /* ... */ }

then I cannot pass this function the second "row" of my array using the syntax: process_sequence(A[1]);

Why can't you just return a const proxy?

Here's the catch: A multi_array has const and nonconst versions of the various operations and I implemented my proxy objects to mimic that behavior. For example, operator[] returns a subarray if the original array is non-const, but a const_subarray if the original array is const. Since I sometimes want the array to allow mutating operations and sometimes not, I cannot merge the const and nonconst versions of these operations in the proxies. I need both const and non-const versions. Thus: subarray const& and const_subarray&

behave alike.

But do they need to? Why not have subarray const& behave like subarray& and const_subarray const& behave like const_subarray& ??

...

class multi_array { const mutable_proxy operator[](int); const immutable_proxy operator[](int); ... };

Now mutable_proxy only needs one version of the array operators. Couldn't that work?

See my comment above.

I don't get it yet.

Also, is one of these member function declarations missing a const qualifier?

The second one.

If so, then this is already what I'm doing and it's not sufficient.

Did you notice the const-qualified return types? Why is it not sufficient?

I know that while working on multi_array, I had some discussions with Jeremy about this and it didn't work for what I was trying to do. That was sometime around 2001 though, so the details are hazy. I've unfortunately had trouble communicating my position, if only because the details quickly fade from my memory each time the issue comes up.

...

I'm not sure; have they used enough imagination?

Good question. I've thought about this on and off for a while and I have yet to come up with or hear of a solution. I'm neither a language lawyer nor a metaprogramming guru, so it's quite possible that I'm overlooking something. :)

Try to answer my questions above and we'll see. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com

Ronald Garcia wrote:

Sorry that it has taken me another week to reply to this one. I've been up to other things and not had time to think about it.

On Dec 14, 2004, at 7:40 AM, David Abrahams wrote:

...

Ronald Garcia wrote:

...

Howdy again,

On Dec 13, 2004, at 10:49 PM, David Abrahams wrote:

...

Ronald Garcia wrote:

...

... Around here (the Open Systems Lab at Indiana University), I've been arguing against this property of C++ for a long time, specifically that the language does not allow you to bind temporary to a non-const reference. I am under the impression that this behavior is meant to "protect" the programmer from making certain mistakes, but the grave downside to this is that it also prevents a library writer from using proxy objects to mimic reference semantics.

The only case I can think of where that's true is when the reference type is the same as the value type. If we allowed operator->

Is this a complete thought, or am I missing something?

I'm still wondering where the above comment was going. What is the rest of the sentence that starts "If we allowed operator->..."?

I think I was going to say that if we allowed it to return a non-const pointer when is_same::value == true, that would handle the cases in question.

...

But do they need to? Why not have

subarray const&

behave like

subarray&

and

const_subarray const&

behave like

const_subarray&

I had thought about this at the time and came to the following conclusion (modulo the fog of time since I made the decision):

From my perspective, it seems that if a template function has a signature such as: template <Container> void my_function(Container& A);

Then that function expects to receive a mutable container type. In the case that a function expects to receive an immutable container, the signature is likely to be:

template <Container> void my_function(Container const& A);

IIRC some templated functions are overloaded as above specifically to handle these two cases.

Yes. Those functions are, in my experience, always used to expose references/pointers that have the right mutability. For example, template <Container> typename Container::value_type& operator[](Container&); template <Container> typename Container::value_type const& operator[](Container const&); They _never_ have different operational semantics. There are several ways to handle that with const proxies. For example, const_subarray<T>::value_type could be the same as subarray<T>::value_type const

I concluded that if I chose the design you describe above, namely that "subarray const&" behave like "subarray&" and "const_subarray const&" behave like "const_subarray&", then I would likely violate the expectations of a template function. Objects of type subarray , the mutable reference proxy, would be passed to the second function above.

I don't think that's a big deal.

Suppose for a moment that a one-dimensional subarray is passed to the above function. Using the above design, A::operator[] would return a mutable reference to an underlying data element, though a const reference to the underlying data might have been expected. That mutable reference could be accidentally mutated (for example, by a similarly overloaded function that mutates in one case and does not in the other).

I don't think so; see above.

In summary, I chose to have subarrays respect the constness built into the language as well as the constness imposed by the choice of proxy object (const_subarray or subarray).

To the best of my memory, this was my thinking at the time.

Understandable, but perhaps this an example of the insufficient imagination I was alluding to. The way I see it, you're asking me to sacrifice the ability to provide const-correctness for convenience, and I'm suggesting that no matter how ugly it may be, there's a way to keep everything safe within the current language. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com

Ronald Garcia wrote:

...

...

...

...

...

> Around here (the Open Systems Lab at Indiana University), I've > been > arguing against this property of C++ for a long time, specifically > that > the language does not allow you to bind temporary to a non-const > reference. I am under the impression that this behavior is meant > to > "protect" the programmer from making certain mistakes, but the > grave > downside to this is that it also prevents a library writer from > using > proxy objects to mimic reference semantics.

The only case I can think of where that's true is when the reference type is the same as the value type. If we allowed operator->

Is this a complete thought, or am I missing something?

I think I was going to say that if we allowed it to return a non-const pointer when is_same::value == true, that would handle the cases in question.

Okay. For multi_array specifically, that doesn't work, because the reference type is a subarray (faux references) but the value type is a multi_array.

...

...

...

But do they need to? Why not have

subarray const&

behave like

subarray&

and

const_subarray const&

behave like

const_subarray&

I had thought about this at the time and came to the following conclusion (modulo the fog of time since I made the decision):

From my perspective, it seems that if a template function has a signature such as: template <Container> void my_function(Container& A);

Then that function expects to receive a mutable container type. In the case that a function expects to receive an immutable container, the signature is likely to be:

template <Container> void my_function(Container const& A);

IIRC some templated functions are overloaded as above specifically to handle these two cases.

Yes. Those functions are, in my experience, always used to expose references/pointers that have the right mutability. For example,

template <Container> typename Container::value_type& operator[](Container&);

template <Container> typename Container::value_type const& operator[](Container const&);

I'm a little confused by this example. When would you pass a container to operator[]?

Those are meant to be free functions and I left out the 2nd argument. Pardon me. Try: Container::value_type& operator[](int); Container::value_type const& operator[](int) const;

Or is that irrelevant to the example you are describing? I am trying to consider functions /that I did not write/ are generic on a container type. The above looks like an implementation detail of a container library to me.

Okay, let's try again: template <class Container> typename Container::value_type& at(Container& c, int i) { if (i < 0 || i > c.size()) throw out_of_range(); return c[i]; } template <class Container> typename Container::value_type const& at(Container const& c, int i); { if (i < 0 || i > c.size()) throw out_of_range(); return c[i]; } My point stands: this kind of overload is always used to expose internal pointers and references with the right constness.

Furthermore, wouldn't an operator[]() return reference type? If reference and value_type are not the same type, does what you describe here still apply?

Sorry that my example was so confusing. But no, you can't use reference here instead of value_type if you are concerned with const-correctness because for any ordinary container C, C and C const have the same ::reference type. Although all of that is irrelevant to the point. My point is that you are saying "overloading on const is important and always returning const proxies breaks that functionality." I am saying there's only one reason people overload on const, and there are other ways to handle that. Generic libraries *would* need to do something special to manage these cases. The 2nd at() signature above won't work. You need something more like: template <class Container> typename mutable_reference<Container>::type at(Container& c, int i) template <class Container> typename constant_reference<Container>::type at(Container const& c, int i); where constant_reference::type is mutable_subarray::value_type& and mutable_reference::type is constant_subarray const&.

...

They _never_ have different operational semantics. There are several ways to handle that with const proxies. For example,

const_subarray<T>::value_type

could be the same as

subarray<T>::value_type const

...

I concluded that if I chose the design you describe above, namely that "subarray const&" behave like "subarray&" and "const_subarray const&" behave like "const_subarray&", then I would likely violate the expectations of a template function. Objects of type subarray , the mutable reference proxy, would be passed to the second function above.

I don't think that's a big deal.

I am presuming that you feel that way because "They _never_ have different operational semantics" right?

Yes.

...

...

Suppose for a moment that a one-dimensional subarray is passed to the above function. Using the above design, A::operator[] would return a mutable reference to an underlying data element, though a const reference to the underlying data might have been expected. That mutable reference could be accidentally mutated (for example, by a similarly overloaded function that mutates in one case and does not in the other).

I don't think so; see above.

Could you elaborate here?

I'm saying that A::operator[] would never return a mutable reference unless A was non-const or was a (const or non-const) mutable_subarray. As a baseline, we'd never violate const correctness.

Are you again referring to "They _never_ have different operational semantics"

No, but that's a good point.

...

Understandable, but perhaps this an example of the insufficient imagination I was alluding to.

Actually I am more worried that my imagination is excessive in certain respects. My concern at the moment is quote I keep referring to above. To be honest I have no example of two such functions having different "operational semantics", by which I assume you mean that the actual text of the two functions is identical, but I can imagine hypothetically that such a case could exist. That's where my imagination may be going too far.

Such a case would be deadly for usability. Imagine: template <class C> void some_algorithm(C& c) { ... some_other_algorithm(c); ... } If some_other_algorithm changed its behavior depending on whether C was const, users of some_algorithm would be very surprised when they passed a const vector.

...

The way I see it, you're asking me to sacrifice the ability to provide const-correctness for convenience, and I'm suggesting that no matter how ugly it may be, there's a way to keep everything safe within the current language.

I seems to me that "keeping everything safe within the current language" relies on the assumption that you make above, that I quote multiple times, and that's a statement not about the language, but about the behavior of programmers, which of course is fine so long as it's true and that no good reason exists for straying from that prescription.

Let's just say that if programmers violate my assumption, returning const rvalue proxies is just going to add another drop to the bucket of confusion that they have already created.

The way I see it, the current ban on binding temporaries to const references is a language level prescription of the same sort you describe above: that it protects people from making so-called "mistakes" at the expense of capabilities that are useful to library developers. I hope that the Move Semantics proposal makes this conversation irrelevant in the end

I have some doubts that it will. After all, people aren't going to stop writing functions that take T& arguments. Even making a proxy movable won't cause it to bind to a T&, IIUC. I think we need a way to express that non-const rvalues of a particular class type should bind to T& (with T deduced as non-const). Perhaps just adding a conversion operator to T& could serve, given the appropriate language extension.

and that the iterator library can take advantage of what it adds to the language. Where can I find a most recent copy of that proposal, btw?

It's in the pre-redmond mailing, in the usual place. You can ask howard if you want more-recent unpublished work.

Since iterator adaptors works just fine today, I'm far more interested in how other aspects of the language will change with respect to these issues. Thanks for the responses. I think that they have somewhat clarified my understanding of the issues.

-- Dave Abrahams Boost Consulting http://www.boost-consulting.com