[intrusive_ptr] Counting implementation

Hello. I want to use intrusive_ptr as a smart pointer to have reference counting without shared_ptr overhead. There's no implementation for embedding reference counting into a pointee class provided by Boost.Smart Ptr library. I'd like to have reference counting easily embedded, like thru inheritance: class test : public intrusive_pointee_base<test> { ... }; I come to an implementation of this. As i'm sure this was coded many times before, two questions: (1) Why stock implementation of intrusive counter is not included to Smart Pointers library ? (2) Is there something wrong with my code ? struct refcount { refcount(void) : count_(0) {} refcount(refcount const&) : count_(0) {} refcount& operator=(refcount const&) {} long operator++(void) { return ::InterlockedIncrement(&count_); } long operator--(void) { return ::InterlockedDecrement(&count_); } private: mutable volatile long count_; }; template<class T> void intrusive_ptr_add_ref(T * p); template<class T> void intrusive_ptr_release(T * p); template<class Derived> class intrusive_pointee_base { friend void intrusive_ptr_add_ref<Derived>(Derived * p); friend void intrusive_ptr_release<Derived>(Derived * p); refcount reference_counter_; protected: ~intrusive_pointee_base(void) {} }; template<class T> void intrusive_ptr_add_ref(T * p) { intrusive_pointee_base<T> * pbase = p; ++(pbase->reference_counter_); } template<class T> void intrusive_ptr_release(T * p) { intrusive_pointee_base<T> * pbase = p; if (!--(pbase->reference_counter_)) delete p; }

AMDG Alexander Gutenev wrote:
I come to an implementation of this. As i'm sure this was coded many times before, two questions: (1) Why stock implementation of intrusive counter is not included to Smart Pointers library ?
No idea. This was requested recently on the devel list, but didn't receive an answer. http://lists.boost.org/Archives/boost/2008/06/139048.php
(2) Is there something wrong with my code ?
It will only compile on windows...
long operator++(void) { return ::InterlockedIncrement(&count_); } long operator--(void) { return ::InterlockedDecrement(&count_);
In Christ, Steven Watanabe

(1) Why stock implementation of intrusive counter is not included to Smart Pointers library ?
No idea. This was requested recently on the devel list, but didn't receive an answer. http://lists.boost.org/Archives/boost/2008/06/139048.php
Thank you. Although the code in that message is more like a pseudocode than an actual implementation, inline non-template friends seems better than templates in my code.
(2) Is there something wrong with my code ?
It will only compile on windows...
long operator++(void) { return ::InterlockedIncrement(&count_); } long operator--(void) { return ::InterlockedDecrement(&count_);
I don't think it's a big problem. If I need to compile on anything else, I'll see the error and use the atomic functions corresponding to those of windows.

This is not an answer to your question, but -- you can take a look at the boost::state_chart events implementation, I think I saw there something like this (search "intrusive_from_this").
I come to an implementation of this. As i'm sure this was coded many times before, two questions: (1) Why stock implementation of intrusive counter is not included to Smart Pointers library ? (2) Is there something wrong with my code ?

This is not an answer to your question, but -- you can take a look at the boost::state_chart events implementation, I think I saw there something like this (search "intrusive_from_this").
Yes, not an answer, but not an answer. Actually there's more than just implementation of counter. However, counting base is implemented as virtual base, not CRTP base, so this is not exactly what I want. I have found something more like what I am trying to implement by searhing entire Boost. it is counted_base in xpressive. However, looks like better to implement it like in Gennadiy's code: struct refcount { refcount(void) : count_(0) {} refcount(refcount const&) : count_(0) {} refcount& operator=(refcount const&) {} long operator++(void) const { return ::InterlockedIncrement(&count_); } long operator--(void) const { return ::InterlockedDecrement(&count_); } private: mutable volatile long count_; }; template<class Derived> class intrusive_pointee_base { friend void intrusive_ptr_add_ref(const Derived * p) { intrusive_pointee_base<Derived> const * pbase = p; ++(pbase->reference_counter_); } friend void intrusive_ptr_release(const Derived * p) { intrusive_pointee_base<Derived> const * pbase = p; if (!--(pbase->reference_counter_)) delete p; } refcount reference_counter_; protected: ~intrusive_pointee_base(void) {} };

Here's the class we use in production:
/** Helper template mixin for @c boost::intrusive_ptr.
To add support for intrusive pointer to class @c T, it should inherit from
@c reference_counter<T>. This will
- provide a reference count member
- force the reference count to initialize to zero
- define the add and release global functions required by @c intrusive_ptr
If this class is not inherited publically, then the class @c T must declare
the global functions as friends to permit them to cast @c T* to @c reference_counter<T>*.
This can be done with the following two lines, with either the @c self typedef
or the string "self" replaced by the actual name of class @c T.
@code
friend void boost::intrusive_ptr_add_ref(self*);
friend void boost::intrusive_ptr_release(self*);
@endcode
@note Due to changes in the C++ standard and design decisions in gcc,
it is no longer possible to declare a template parameter as a friend class.
(Basically, you can't use a typedef in a friend declaration and gcc treats
template parameters as typedefs).
@note You can use this with insulated (by name only) classes. The important
thing is to make sure that any class that uses an @c intrusive_ptr to a
by name only class has all of its constructors and destructors declared
in the header and defined in the implementation translation unit. If the
compiler generates any of those, it will not compile due to missing
functions or methods.
*/
template <typename T> class reference_counter
{
protected:
typedef reference_counter self; //!< Useful for self type reference
//! Default constructor.
reference_counter() : m_reference_count(0) { }
/** Copy constructor.
@note Always initialize to zero. Ignore the source count.
This makes it nicer for clients who can now use the default
copy constructor.
*/
reference_counter(self const&) : m_reference_count(0) { }
/** Assignment operator.
@note Prevent the count from being assigned.
*/
self& operator= (self const&) { return *this; }
int m_reference_count; //!< The reference count
//! Define the helper functions and give them access
//!@{
friend inline void intrusive_ptr_add_ref(T* t) { ++(static_cast

Compared to mine, better sides: 1. All in single class (I have now joined them too). 2. Assignment operator returns correct reference. 3. typedef made code cleaner. Worse sides: 1. No thread safety (may be you do not need it, but i need). 2. No const awareness, i think the following will fail: boost::intrusive_ptr<const test> p(new test); boost::intrusive_ptr<const test> q(new test); p = q; 3.m_reference_count is visible to derived classes.

At 04:57 PM 7/30/2008, you wrote:
Compared to mine, better sides: Worse sides: 1. No thread safety (may be you do not need it, but i need). 2. No const awareness, i think the following will fail: boost::intrusive_ptr<const test> p(new test); boost::intrusive_ptr<const test> q(new test); p = q; 3.m_reference_count is visible to derived classes.
1) Yes, but as you note it's trivial to add if it's important. I had that in my own implementation of intrusive_ptr (before those were in Boost) but dropped it because I never used it. Not such a big deal on Win32, but I remember it being painful on Sun / Sparc systems (am I dating myself?). 2) Interesting. I have been using this code for years and not had that come up. I'll have to take a look. 3) A design feature for me. It makes copy on write semantics easier to implement. You might consider putting in "bool is_shared() const { return m_reference_counter > 1; }" for that case.

1) Yes, but as you note it's trivial to add if it's important. I had that in my own implementation of intrusive_ptr (before those were in Boost) but dropped it because I never used it. Not such a big deal on Win32, but I remember it being painful on Sun / Sparc systems (am I dating myself?).
boost::shared_ptr have reference counting for various platforms, and there's no policy to have custom reference counter, so, although I've never coded for Sun / Sparc systems, it seems possible to do what boost::shared_ptr does.
3) A design feature for me. It makes copy on write semantics easier to implement. You might consider putting in "bool is_shared() const { return m_reference_counter > 1; }" for that case.
I would rather put "bool is_shared() const { return m_reference_counter > 1; }" as protected member, and m_reference_counter as private one. Do you need it for something like "intrusive_from_this()" form Boost.Statechart ?

At 08:26 AM 7/31/2008, you wrote:
I would rather put "bool is_shared() const { return m_reference_counter > 1; }" as protected member, and m_reference_counter as private one. Do you need it for something like "intrusive_from_this()" form Boost.Statechart ?
I haven't used Boost.Statechart so I can't speak to that. I do lot of psuedo-value classes, which are PIMPL classes based on intrusive_ptr that do copy on write to act like values instead of handles when it matters (Qt from Trolltech does a lot of that, for a public example). That has an obvious need to test for the reference count being larger than 1. Rather than is_shared(), you might duplicate the shared_ptr interface, with unique() and use_count(). That'd probably be better for something including in Boost.

Updated, after looking into other's implementation:
template<class Derived>
class intrusive_pointee_base
{
private:
typedef intrusive_pointee_base self;
protected:
intrusive_pointee_base(void) : reference_counter_(0) {}
intrusive_pointee_base(self const&) : reference_counter_(0) {}
intrusive_pointee_base operator=(self const&) { return *this }
~intrusive_pointee_base(void) {}
private:
friend void intrusive_ptr_add_ref(const Derived * p)
{
::InterlockedIncrement(&static_cast

Alexander Gutenev
Updated, after looking into other's implementation:
I would add IncrementPolicy with following default. struct trivial_increment_policy { void inc( long& c ) { ++c; } bool dec( long& c ) { return --c == 0; } }
template<class Derived> template
class intrusive_pointee_base { private: typedef intrusive_pointee_base self; protected: intrusive_pointee_base(void) : reference_counter_(0) {} intrusive_pointee_base(self const&) : reference_counter_(0) {} intrusive_pointee_base operator=(self const&) { return *this } ~intrusive_pointee_base(void) {}
private: friend void intrusive_ptr_add_ref(const Derived * p) { IncrementPolicy::inc(&static_cast
(p)->reference_counter_); } friend void intrusive_ptr_release(const Derived * p) { if(!IncrementPolicy::dec( &static_cast<self const>*>(p)->reference_counter_)) delete p; }
mutable volatile long reference_counter_; };
Now you can add your own struct nt_mt_safe_increment { void inc( long& c ) { ::InterlockedIncrement(c); } bool dec( long& c ) { return ::InterlockedDecrement(--c) == 0; } }; Gennadiy

I would add IncrementPolicy with following default.
struct trivial_increment_policy { void inc( long& c ) { ++c; } bool dec( long& c ) { return --c == 0; } }
Now you can add your own
I considered this, but for some reason there's no policies in shared_ptr, so
I decided not to add one too.
I think counter should belong to policy too (but no-copying is still up to
intrusive_pointee_base).
struct trivial_increment_policy {
void inc() { ++c; }
bool dec() { return --c == 0; }
int c;
}
template

I did not do that, because shared_ptr does not.
Counter should be a part of policy too, like:
struct trivial_increment_policy {
void inc( ) { ++c; }
bool dec( ) { return --c == 0; }
size_t c;
}
template

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday 30 July 2008 16:03 pm, Alexander Gutenev wrote:
I want to use intrusive_ptr as a smart pointer to have reference counting without shared_ptr overhead.
I'm curious, could you be more specific about how your implementation has less overhead than shared_ptr? Do you mean it uses less memory? Compiles faster? Runs faster? -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFIkcyp5vihyNWuA4URAq6vAKDKntmBVbQadV0WR6drrZ+WssqzsgCg2liN 5D4WJvBFZq6CmbrLQ1FFysQ= =2604 -----END PGP SIGNATURE-----

On Wednesday 30 July 2008 16:03 pm, Alexander Gutenev wrote:
I want to use intrusive_ptr as a smart pointer to have reference counting without shared_ptr overhead.
I'm curious, could you be more specific about how your implementation has less overhead than shared_ptr? Do you mean it uses less memory? Compiles faster? Runs faster?
Less memory. The overhead of shared_ptr is allocation of another object which has reference count and deleter object. It is allocated in a separate block, hence memory overhead - that's the most noticeable when I have a lot of pointers to small objects. Also shared_ptr itself is double sized then. Also it is a bit more slow for allocation and deletion - but this is not noticeable in my case.

Alexander
On Wednesday 30 July 2008 16:03 pm, Alexander Gutenev wrote:
I want to use intrusive_ptr as a smart pointer to have reference counting without shared_ptr overhead.
I'm curious, could you be more specific about how your implementation has less overhead than shared_ptr? Do you mean it uses less memory? Compiles faster? Runs faster?
Less memory. The overhead of shared_ptr is allocation of another object which has reference count and deleter object. It is allocated in a separate block, hence memory overhead - that's the most noticeable when I have a lot of pointers to small objects.
You confuse thing a bit I believe. The over head is not caused so much by the extra object itself as by the pointer to this object that have to reside inside every shared_ptr doubling it's size effectively. Gennadiy

You confuse thing a bit I believe. The over head is not caused so much by the extra object itself as by the pointer to this object that have to reside inside every shared_ptr doubling it's size effectively.
Gennadiy
It depends, I think. Assume shared_ptr is used as an easy way to deal with polymorphic objects, like putting them into std::vector. Then most of the pointers would be unique. So vector data size is doubled, but this is not the most overhead. The most overhead is counter and deleter objects, those objects are larger than a single pointer, and each of them is stored in dedicated heap block. If shared_ptr is used really to share objects, then doubling pointer size may cause more overhead than counter and deleter.

Alexander Gutenev
You confuse thing a bit I believe. The over head is not caused so much by the extra object itself as by the pointer to this object that have to reside inside every shared_ptr doubling it's size effectively.
Gennadiy
It depends, I think. Assume shared_ptr is used as an easy way to deal with polymorphic objects, like putting them into std::vector. Then most of the pointers would be unique. So vector data size is doubled, but this is not the most overhead. The most overhead is counter and deleter objects, those objects are larger than a single pointer, and each of them is stored in dedicated heap block. If shared_ptr is used really to share objects, then doubling pointer size may cause more overhead than counter and deleter.
Overhead of counter and deleter is exactly the same for both shared_ptr and intrusive_ptr - one per object (the fact that intrusive_ptr doesn't support Deleter at the moment is beside the point IMO - it should). Slight difference is that former is require additional new call. The only "real" advantage of intrusive_ptr is the size of it's instances Gennadiy

Overhead of counter and deleter is exactly the same for both shared_ptr and intrusive_ptr - one per object (the fact that intrusive_ptr doesn't support Deleter at the moment is beside the point IMO - it should). Slight difference is that former is require additional new call.
Additional new call is not very little thing. Extra heap operation have its cost. Not sure for other platforms, but with VS2005, new is roughly malloc, malloc is roughly HeapAlloc, and Windows heap have overhead like rounding up to multiply of 16 bytes per HeapAlloc and using additional 16 bytes per HeapAlloc. The reason for polymorphic deleter in shared_ptr is, IMHO, that additional heap object is allocated anyway, so here polymorphic deleter is almost free sugar. For unique_ptr deleter is non-polymorphic, unique_ptr does not need additional object by itself, so here polymorphic deleter would be an expensive thing. For intrusive_ptr polymorphic deleter is too expensive too. As for non-polymorphic - this can be controlled by intrusive_ptr_add_ref implementation. Also, there's an alternative for custom non-polymorphic deleter: member operator delete. For PODs or not editable classes it cannot be added, but then inheritance from intrusive_pointee_base cannot be added too. If class is yours, why not write operator new/operator delete for it ?

Overhead of counter and deleter is exactly the same for both shared_ptr and intrusive_ptr - one per object (the fact that intrusive_ptr doesn't support Deleter at the moment is beside the point IMO - it should). Slight difference is that former is require additional new call.
The only "real" advantage of intrusive_ptr is the size of it's instances
Gennadiy
Overhead of counter and deleter not the same because of heap allocation overhead. On VS2005 new is roughly malloc, and malloc is roughly HeapAlloc. HeapAlloc allocation overhead is like extra 16 bytes per allocation plust rounding up to 16 bytes. I'm not optimizing prematurely, allocations were spotted by profiler. For shared_ptr, polymorhic deleter adds alomst no overhaed, because reference counter is allocated as extra block anyway, so polymorhic deleter is sugar for free. For unique_ptr nonpolymorhic deleter is used, because no extra block is allocated, so it would be too expensive to have extra block dedicated to polymothic deleter. For intrusive_ptr polymorhic deleter is too expensive too, and nonpolymorhic can easily be added by intrusive_ptr_add_ref implementetion. Also, member operator delete can be used to control deletion. Of course it cannot be used for PODs, or classes that cannot be changed, but when you can inherit from intrusive_pointee_base, you can also add new/delete operators.

On Wednesday 30 July 2008 16:03 pm, Alexander Gutenev wrote:
I want to use intrusive_ptr as a smart pointer to have reference counting without shared_ptr overhead.
I'm curious, could you be more specific about how your implementation has less overhead than shared_ptr? Do you mean it uses less memory? Compiles faster? Runs faster?
There's several ways to allocate reference counter: 1. In separate object allocated separately - this is shared_ptr 2. In the same buffer with object but not as a part of object - this is shifted_ptr 3. In the pointed object as a part of object - this is intrusive_ptr the way i'm trying to use it. The posted code is an implementation of what goes to the object. It actually not even necessary for a shared pointer to use an integer reference counting.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday 31 July 2008 11:53 am, Alexander wrote:
There's several ways to allocate reference counter: 1. In separate object allocated separately - this is shared_ptr 2. In the same buffer with object but not as a part of object - this is shifted_ptr
I'm getting a little off-topic, but actually shared_ptr supports #2 as well if you create it with make_shared/allocate_shared.
3. In the pointed object as a part of object - this is intrusive_ptr the way i'm trying to use it. The posted code is an implementation of what goes to the object. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFIkedQ5vihyNWuA4URAlXyAKDi9NABQSHBevqHkIgo+Td2oNYsfgCg60P2 abs9DBlxWouVOTQ51oBuX8Q= =9Hhi -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday 31 July 2008 15:28 pm, Alexander Gutenev wrote:
I'm getting a little off-topic, but actually shared_ptr supports #2 as well if you create it with make_shared/allocate_shared.
Interesting. Is this in Boost already ?
It looks like boost/make_shared.hpp will be in 1.36, although documentation for it won't be. I put a patch that adds some documentation in trac ticket #1897, but I don't think anything has been done with it. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFIkh6x5vihyNWuA4URAuB5AJ4mOpagrvZQYZOyEkXbkh4jn8HuGACgoLgo 1xnGwwSxHJeGIsnailHY8CQ= =hgup -----END PGP SIGNATURE-----
participants (7)
-
Alan M. Carroll
-
Alexander
-
Alexander Gutenev
-
Frank Mori Hess
-
Gennadiy Rozental
-
Igor R
-
Steven Watanabe