[valid_ptr] proposing a new type of pointer object that allows validity tracking without memory management and object ownership

Dear all, I am posting to propose a new light weight pointer object library (boost.valid_ptr) that provides safe validity checking without having to take ownership or the the responsibility of memory management of the object pointed to. The intention is to develop a useful pointer object that is easy to use and handles validity in a simple and invisible manner, to be used just like a normal pointer and in a way that a programmer would be familiar with, to produce cleaner and more intuitive code than other pointer libraries for its problem domain. The library would be best used when it is difficult to determine when an object will be released, for example, when pointing to objects higher on the stack, objects that are members of other objects or objects contained within a boost.ptr_container. The library is to be practically useful and most useful where boost.smart_ptr and boost.ptr_container libraries are not appropriate (for example, due to memory management or ownership not being required). The valid_ptr library is to be optimized for access rather than fast assignment / de-assignment. Detailed proposal: https://docs.google.com/document/pub?id=1wod4CbeL9x8rLJ2S8815akMAGmYj-fCq90T... I am posting here in the hope it will generate suitable interest that it may be worth developing as a full library for submission. Best wishes, Dan

On 03/16/2011 01:57 PM, Dan Walters wrote:
Dear all,
I am posting to propose a new light weight pointer object library (boost.valid_ptr) that provides safe validity checking without having to take ownership or the the responsibility of memory management of the object pointed to.
The intention is to develop a useful pointer object that is easy to use and handles validity in a simple and invisible manner, to be used just like a normal pointer and in a way that a programmer would be familiar with, to produce cleaner and more intuitive code than other pointer libraries for its problem domain.
The library would be best used when it is difficult to determine when an object will be released, for example, when pointing to objects higher on the stack, objects that are members of other objects or objects contained within a boost.ptr_container.
The library is to be practically useful and most useful where boost.smart_ptr and boost.ptr_container libraries are not appropriate (for example, due to memory management or ownership not being required).
The valid_ptr library is to be optimized for access rather than fast assignment / de-assignment.
Detailed proposal: https://docs.google.com/document/pub?id=1wod4CbeL9x8rLJ2S8815akMAGmYj-fCq90T...
I am posting here in the hope it will generate suitable interest that it may be worth developing as a full library for submission.
Best wishes,
Dan
Hello, how and when is this better than what can be achieved by using intrusive_ptr<T> and its associated function overloads? Cheers, Rutger

On 16/03/2011 13:57, Dan Walters wrote:
Dear all,
I am posting to propose a new light weight pointer object library (boost.valid_ptr) that provides safe validity checking without having to take ownership or the the responsibility of memory management of the object pointed to.
The intention is to develop a useful pointer object that is easy to use and handles validity in a simple and invisible manner, to be used just like a normal pointer and in a way that a programmer would be familiar with, to produce cleaner and more intuitive code than other pointer libraries for its problem domain.
The library would be best used when it is difficult to determine when an object will be released, for example, when pointing to objects higher on the stack, objects that are members of other objects or objects contained within a boost.ptr_container.
The library is to be practically useful and most useful where boost.smart_ptr and boost.ptr_container libraries are not appropriate (for example, due to memory management or ownership not being required).
It seems very similar to what weak_ptr does, except it requires the object to be allocated through valid_wrapper rather than with shared_ptr. I like it because it's more lightweight though.

Dan Walters wrote:
I am posting to propose a new light weight pointer object library (boost.valid_ptr) that provides safe validity checking without having to take ownership or the the responsibility of memory management of the object pointed to.
...
Detailed proposal: https://docs.google.com/document/pub?id=1wod4CbeL9x8rLJ2S8815akMAGmYj-fCq90T...
This only works in single-threaded code, and even then, it can fail. In MT code, it's possible that the object is destroyed after you called get() and before you had the opportunity to look at the result (which is now a dangling pointer). In ST code, things get interesting when in: valid_ptr<X> px; px->f(); f() indirectly causes the object to be destroyed. This happens more often than one might think. :-) weak_ptr requires one to create a shared_ptr not because I wanted things to be less efficient, but because this shared_ptr keeps the object alive; this way, when you receive a positive answer from weak_ptr::lock, you can be sure it stays positive. Unless you're using null deleters of course, but the risk is all yours then. :-)

Dear all, Rutger, Mathias, Peter, Matt, Joël Lamotte, thanks for your replies, I appreciate your input. Rutger, as I understand, intrusive_ptr<T> is a reference counting type pointer that takes shared responsibility for the memory allocation of the object it points to. Thus, the object pointed to cannot be destroyed if an intrusive_ptr to it still exists. With valid_ptr, a large part of the concept is that the programmer doesnt always want to control when the object can be destroyed. However, the programmer does always wants to know if the pointer points to a valid piece of memory. Valid_ptr is more like a normal c++ pointer than a smart pointer. It assumes no ownership of the object, it simply points to the memory. It's only feature up from a normal c++ pointer is that it can check that the object it points to is still valid memory. Matt, Valid_ptr would do a similar thing to using shared_ptr and weak_ptr. Valid_ptr could be better when very frequent access is required (very little access overhead while weak_ptr requires a little more) and it can also be less intrusive to the code as it doesnt require the code to relinquish control of memory allocation (should the programmer have his own strategy). Sometimes valid_ptr would give cleaner code, it is lighter weight and much more primitive under the hood. But shared_ptr and weak_ptr have more features and have many advantages in other cases. Perhaps weak_ptr and valid_ptr are similar in the way std::list and std::vector are similar. Similar functionality, but one is usually better than the other depending on how they are used. I understand the key difference to be that valid_ptr does not imply ownership, sometimes an advantage, sometimes disadvantage. I think the problem domain I am defining is distinctly different to that for which shared_ptr and weak_ptr is designed - valid_ptr is not concerned with remembering to delete the memory it points to, or take responsibility for this. It is only for keeping track of if the object being pointed to has been destroyed or not. I consider it a lighter weight object that is somewhere between a normal c++ pointer and a smart ptr. Peter, I agree, I am yet to come up with a MT solution that makes sense here, that needs thought. I do not understand your comment about f() indirectly causing the object to be destroyed, could you explain this a little more? if f() is a member function, surely that would involve the line "delete this;" ? I am missing something... I appreciate how strong a library the smart_ptrs are and understand the reasoning behind locking weak_ptrs in order to access them. As you explain, this is particularly awesome in a MT environment. I did comment that weak_ptr is slightly heavier weight for accessing it's value (than to valid_ptr) but appreciate that it is essential in order to ensure the object isnt destroyed in a MT environment. What I am proposing is a ptr object that doesnt consider this by design - greater performance (for accessing) (but not safe in a MT environment?). My understanding is that valid_ptr fills the gap between c++ pointer and smart_ptr. Essentially, it is just a normal c++ pointer with a thin layer of underlying code to help stop the programmer from accessing an invalid pointer. Regarding the safety issues, the valid_ptr is not as fool proof as a smart_ptr (because it doesnt manage memory assignments), but it is safer than a normal c++ pointer and used in a very similar way so maybe this is ok. Matt, Your passive_ptr concept differs from valid_ptr but your reference to semantic intent is very significant. I think that valid_ptr implies that the object being pointed to is expected to become invalid at some point, and that it is not safe to use a regular pointer. I think it also implies that the object being pointed to is owned by another object or container and not under the control of valid_ptr. An important point to understand that valid_ptr is only half of the library - the other half being valid_target. The target object is inherited or wrapped to provide a reference register (a lot like intrusive_ptrs reference count). When the object is destructed, the reference register is used to inform all the valid_ptrs pointing to that object that they are no longer valid. That is how validity is ensured. It results in light weight valid_ptrs although the target object may be heavier weight. Joël Lamotte, This is not correct. I am sorry, I have not explained this clearly enough. As stated above, the valid_ptr DOES work just like a raw pointer. It is the object that is pointed to that does all the hard work. The valid_ptr can subscribe and un-subscribe to the object it points to. The subscription list that is stored on the object that is being pointed to is used to reset all subscribed pointers when the object is destroyed. Hence, when the pointer is assigned, its value goes from NULL to the value of that object. The object also receives a subscription and knows which pointers are pointing to it. When the object is destroyed, its dtor resets all the valid_ptrs that are subscribed. Matt, I am unsure if I follow your second post. Perhaps what I have mentioned above answers this? Thank you again to all who have been kind enough to give feedback, it is much appreciated. Best wishes, Dan

Dan Walters wrote:
I do not understand your comment about f() indirectly causing the object to be destroyed, could you explain this a little more? if f() is a member function, surely that would involve the line "delete this;" ? I am missing something...
It's rarely as direct as "delete this"; the chain is usually more convoluted, f.ex. p->f() calls g() which calls h() which happens to replace the contents of a container that owns the object.

Since the MT issues had already been raised, both of my comments were concerning similar-but-different cases than what you proposed. passive_ptr<> would simply indicate non-ownership, without making any assertions about object lifetime or pointer validity. The comment about nonnull_ptr<> was just riffing on Joël's comment. It might be interesting to explore the possibility of templates which parameterize both the object lifetime management policy and pointer validity (or non-null-ness, at least). Whether it would be worthwhile is a separate question. It does seem to me that one solution you might use to solve the MT problems would be to put a shared_mutex in valid_target<> and hide some locking in accesses via valid_ptr<> (i.e. put a scoped_lock in a temporary object returned by valid_ptr<>'s operators). Then, valid_target<> acquires an exclusive lock, when it's directed to delete the referenced object. Unfortunately, I don't see how that would be better than weak_ptr<>. It certainly wouldn't be any faster or lighter-weight. I can see how it would solve certain shared-ownership scenarios where you don't know, a priori, who will delete a shared object or when. But the same thing could be accomplished by using weak pointers to an object whose refcount had been manually incremented by 1. When you want to kill it, one of the referers could then manually decrement the refcount, so that it gets killed when it goes back to idle. Matt -----Original Message----- From: boost-bounces@lists.boost.org on behalf of Dan Walters Sent: Wed 3/16/2011 9:19 PM To: boost@lists.boost.org Subject: Re: [boost] [valid_ptr] proposing a new type of pointer object that allows validity tracking without memory management and object ownership
Matt,
I am unsure if I follow your second post. Perhaps what I have mentioned above answers this?
Thank you again to all who have been kind enough to give feedback, it is much appreciated.
Best wishes,
Dan

Peter, I understand now. That is a brilliant feature of the shared_ptr design that allows an object to release itself while not being fully released until its member function returns (when the shared_ptr on the stack goes of scope). However, with a normal pointer this would not be safe - it would be very bad practice for an object that was not wrapped by a shared_ptr to release itself - this would result in an invalid this pointer. Locking the object before accessing is useful / essential when the object may want to destroy itself or even just to avoid run time errors when the programmer makes a mistake. However, I dont think it is necessarily a bad thing to forgo locking when the object is not meant to be able to release itself by design (in a ST environment). Valid_ptr is no where near as smart as a shared_ptr/weak_ptr, it works much more like a normal pointer, it takes a passive relationship to the object it points to, resulting in a more passive design. I think there are strong cases where the strengths of shared_ptr/weak_ptr are just not required and something between a normal and smart pointer is required.

On 17 March 2011 03:33, Gruenke, Matt <mgruenke@tycoint.com> wrote: ... the possibility of templates which parameterize both the object lifetime management policy and pointer validity (or non-null-ness, at least). Whether it would be worthwhile is a separate question. I think the issue of semantics, showing intent, suggesting behavior is very relevant. Perhaps the name subscribed_ptr describes the usage and mechanics better than valid_ptr does? Matt <mgruenke@tycoint.com> wrote: ... put a shared_mutex in valid_target<> and hide some locking in accesses via valid_ptr<> (i.e. put a scoped_lock in a temporary object returned by valid_ptr<>'s operators) ... Unfortunately, I don't see how that would be better than weak_ptr<> Right now, I agree. I think thread safety is beyond the ambitions of valid_ptr which should be intentionally light weight. On the flip side, there are plenty of MT programs out there that don't use reference counting smart pointers and get by fine. Matt <mgruenke@tycoint.com> wrote: the same thing could be accomplished by using weak pointers to an object whose refcount had been manually incremented by 1. When you want to kill it, one of the referers could then manually decrement the refcount, so that it gets killed when it goes back to idle. valid_ptr should not be able to release it's target object by design - it is not a memory management pointer.

-----Original Message----- From: boost-bounces@lists.boost.org on behalf of Dan Walters Sent: Thu 3/17/2011 8:21 AM To: boost@lists.boost.org Subject: Re: [boost] [valid_ptr] proposing a new type of pointer object that allows validity tracking without memory management and object ownership On 17 March 2011 03:33, Gruenke, Matt <mgruenke@tycoint.com> wrote: ...
Matt <mgruenke@tycoint.com> wrote: the same thing could be accomplished by using weak pointers to an object whose refcount had been manually incremented by 1. When you want to kill it, one of the referers could then manually decrement the refcount, so that it gets killed when it goes back to idle.
valid_ptr should not be able to release it's target object by design - it is not a memory management pointer.
It doesn't matter whether valid_ptr can be used to delete the object. The point was that in order to fulfill the contract of ensuring that the pointer is valid while used, in a threadsafe fashion, it would need to either hold a lock or a reference count preventing the object from being deleted while an expression involving one of the access operators is being evaluated. As you point out, valid_ptr doesn't need to solve the problem of thread synchronization in order to be usable in a multithreaded environment. However, if it doesn't, the question is whether it provides enough value that people would use it. Matt

without memory management and object ownership
I think a useful complement to the pointer types in boost.smart_ptr would be something I've referred to as passive_ptr<>. It's passive in the sense that it doesn't imply object ownership and therefore doesn't involve itself in object lifetime management. The problem I am interested in solving is the case where you see bare pointers (i.e. non-smart pointers) being used in code and you're not sure whether there's a good reason that they're not one of the smart pointers (or auto_ptr<>, which I normally use in cases where I really want scoped_ptr<>, but I need to transfer ownership out of it). Also, it's helpful to easily know that it's not a bug that the pointer isn't deleted. In other words, a passive_ptr<> class wouldn't add any functionality over a bare pointer, but it would enable users to communicate semantic intent. Perhaps that covers some of what you're targeting with valid_ptr<>? As for ensuring validity, I don't see how this is possible without getting involved in lifetime management. Matt

As far as I understand, it's only a type working like a raw pointer but with checks on assignation and copy. Those checks will only assume that the pointer don't have a NULL value, (or maybe points to an adress that i'snt possible?) is it correct understanding of this library? Joël Lamotte.

"Non-NULL" is different than "valid". Maybe such a thing should be called nonnull_ptr<>? In the case of passive_ptr<>, I do allow it to be NULL. While a nonnull_ptr<> would useful, it would only cover some of the cases in which I use passive_ptr<>. I don't see why you'd have to check for NULL upon copy or assignment from another nonnull_ptr<>. It only needs to check in the constructor and when the value is directly changed. In fact, aside from that runtime check and the ability to alter the value, I don't really see what it buys you over using a reference. Also, whether the value is NULL is orthogonal to ownership - I could imagine having a nonnull_shared_ptr<>, nonnull_scoped_ptr<>, etc. Matt -----Original Message----- From: Klaim - Joël Lamotte [mailto:mjklaim@gmail.com] Sent: Wed 3/16/2011 6:37 PM To: boost@lists.boost.org Cc: Gruenke, Matt Subject: Re: [boost] [valid_ptr] proposing a new type of pointer object that allows validity tracking without memory management and object ownership As far as I understand, it's only a type working like a raw pointer but with checks on assignation and copy. Those checks will only assume that the pointer don't have a NULL value, (or maybe points to an adress that i'snt possible?) is it correct understanding of this library? Joël Lamotte.
participants (6)
-
Dan Walters
-
Gruenke, Matt
-
Klaim - Joël Lamotte
-
Mathias Gaunard
-
Peter Dimov
-
Rutger ter Borg