Any help on this? On Tue, Jun 23, 2009 at 3:20 PM, Robert Dailey <rcdailey@gmail.com> wrote:

I forgot to mention that I'm on Windows. But this, of course, would need to be portable.

On Tue, Jun 23, 2009 at 3:16 PM, Robert Dailey <rcdailey@gmail.com> wrote:

...

Hi, Suppose I have a DLL that allocates a few objects and places them in a ptr_vector. I then pass this ptr_vector to my executable via a move operation (basically a swap, not a copy). The executable would keep the ptr_vector by value and eventually delete it when it falls out of scope.

Is there any danger in doing this? My goal is to make sure that the same memory manager is used to delete the items as was used to create them.

On Thu, Jun 25, 2009 at 9:41 AM, Thorsten Ottosen < thorsten.ottosen@dezide.com> wrote:

Do you think a stateful custom clone_allocator would allow you to do what you want?

Could you elaborate a bit on what you mean? What state would you give to clone_allocator? Keep in mind that clone_allocator needs to, at some point, be implemented IN the DLL itself, otherwise the correct 'delete' will not be called. Some sort of polymorphism would solve this, that's the most obvious solution I can think of. Either that, or instead of a functor, take a boost::function, and that way I can pass you a function pointer to call for clone allocation/deallocation. There's several easy ways of solving this particular issue that I can see. But I would be interested in hearing more about the idea you have. I'm not sure I fully understand just yet.

How does a clone_allocator make a new clone, if it needs access to the allocator in the parent ptr_container? I had this problem when I tried to write a clone allocator for monotonic to support ptr_containers: struct inline_clone_allocator { template< class U > static U* allocate_clone( const U& r ) { // can't allocate clone without access to the allocator?? return 0; } What am I missing? On Fri, Jun 26, 2009 at 6:05 AM, Thorsten Ottosen < thorsten.ottosen@dezide.com> wrote:

Robert Dailey skrev:

...

On Thu, Jun 25, 2009 at 9:41 AM, Thorsten Ottosen < thorsten.ottosen@dezide.com> wrote:

...

Do you think a stateful custom clone_allocator would allow you to do what you want?

Could you elaborate a bit on what you mean? What state would you give to clone_allocator?

That is up to you. The important fact is that allocator is stored as an object, and/or that you can access and modify that object to your wishes.

Keep in mind that clone_allocator needs to, at some point,

...

be implemented IN the DLL itself, otherwise the correct 'delete' will not be called. Some sort of polymorphism would solve this, that's the most obvious solution I can think of. Either that, or instead of a functor, take a boost::function, and that way I can pass you a function pointer to call for clone allocation/deallocation. There's several easy ways of solving this particular issue that I can see.

But I would be interested in hearing more about the idea you have. I'm not sure I fully understand just yet.

Let me give you an example of what I was trying to do. To avoid storing duplicates of NullObjects, I want to let my clone_allocator

1. don't delete a null object

2. clone a null object by returning just a pointer (i.e. don't allocate anything)

This is how I could do that:

template<class NullObject> class null_object_clone_allocator { public: typedef NullObject null_object_type;

private: const NullObject* null_object_;

public: explicit null_object_clone_allocator( const NullObject& x ) : null_object(&r) { }

template< class U > U* allocate_clone( const U& r ) const { if( &r == null_object_ ) return null_object_; return new_clone( r ); }

template< class U > void deallocate_clone( const U* r ) const { if( r != null_object_ ) delete_clone( r ); } };

boost::ptr_vector<base,null_object_clone_allocator<null_base>> vec( my_alloc) ;

-Thorsten

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

If you want to provide a stateful clone allocator, it would allow me to do something like this: class clone_allocator { public: clone_allocator() { dll_clone_object = GetProcAddress(); } MyClass* operator() ( MyClass const& node ) { return dll_clone_object( node ) } private: MyClass* (*dll_clone_object)( MyClass const& ); }; Note that the above is pseudocode. The point is, it would allow me to abstract the DLL factory function for creating clones, thus giving me full control over which 'delete' and 'new' method is called (In this case, it would be the one from the DLL's memory manager). On Thu, Jun 25, 2009 at 4:26 PM, Thorsten Ottosen < thorsten.ottosen@dezide.com> wrote:

Christian Schladetsch skrev:

...

How does a clone_allocator make a new clone, if it needs access to the allocator in the parent ptr_container? I had this problem when I tried to write a clone allocator for monotonic to support ptr_containers:

struct inline_clone_allocator { template< class U > static U* allocate_clone( const U& r ) { // can't allocate clone without access to the allocator?? return 0; }

What am I missing?

Nothing, I think. This seems like something it was not designed for, on the cnotrary the clone is currently always allocated by someone else.

The current design uses static functions, and so there can be no stateful allocator.

I thought a little about this, but have not come up with a solution. For example, how do we design an allocator that knows how to clone a class hierarchy? Only the class itself knows how to clone itself because it knows the exact type. OTOH, the classes in the class hieararchy doesn't know about the allocator.

We might try a new way to clone objects by replacing

virtual Base::clone() const = 0;

with

virtual Base::clone( boost::clone_allocator& ) const = 0;

Now, if the containers stored a clone_allocator instance, it could be implemented like

struct my_clone_allocator : boost::clone_allocator { template< class U > U* allocate_clone( const U& r ) { return r.clone( *this ); } };

(remark: boost::clone_allocator would have two virtual functions: void* allocate(unsigned) and void deallocate(void*)).

That might work, but it still gives you no guarantee or possibility that derived classes use the container's allocator for all its data members (reqursively). Maybe that is not required; I just don't know.

So feedback on what we want to do is most welcome.

-Thorsten _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

If you want to provide a stateful clone allocator, it would allow me to do

...

something like this:

If ptr_container<T> is to have stateful allocators, should it not use boost::container<T>? Currently the ptr_containers are based on the std::containers. Is there a case against using boost::container instead? Other than that Ion has not committed the new headers yet of course... Regards, Christian

On Fri, Jun 26, 2009 at 9:26 AM, Thorsten Ottosen < thorsten.ottosen@dezide.com> wrote:

Christian Schladetsch skrev:

...

How does a clone_allocator make a new clone, if it needs access to the allocator in the parent ptr_container? [snip]

What am I missing?

Nothing, I think. This seems like something it was not designed for, on the cnotrary the clone is currently always allocated by someone else.

Then what is the point of clone_allocator?

The current design uses static functions, and so there can be no stateful allocator.

I thought a little about this, but have not come up with a solution. For example, how do we design an allocator that knows how to clone a class hierarchy? Only the class itself knows how to clone itself because it knows the exact type. OTOH, the classes in the class hieararchy doesn't know about the allocator.

We might try a new way to clone objects by replacing

virtual Base::clone() const = 0;

Isn't this what a copy constructor is for? I've had a quick look at <boost/ptr_container/ptr_inserter.hpp> and related files, and it seems that whenever it currently says something like obj = container_type::clone_allocator_type::allocate_clone(*r); it could say instead obj = container_type::clone_allocator_type::allocate_clone(*r, cont.get_allocator()); And then in a custom clone_allocator, you could say: struct my_clone_allocator { template< class U, class Allocator > static U* allocate_clone( const U& r, Allocator &alloc) { typename Allocator::template rebind<U>::other my_alloc(alloc); U *clone = my_alloc.allocate(1); my_alloc.construct(clone, r); return clone; } } Or, pass in a rebound allocator to allocate_clone; it doesn't really matter. Point being, can you just pass the container's allocator to the clone_allocator? Another alternative is to make clone_allocator a concept and pass the either-or-both U and Allocator types to it at compile time, but I think it works just as well and more generally as a static method. Regards, Christian.

[snip]

...

Then what is the point of clone_allocator?

It does provide some room for customization. But it was deliberately kept very conservative, so we can extend it when we need to in light of user experience.

Isn't being able to create a clone using the parent containers' allocator a practical necessity? The originals are made using that allocator - isn't it natural to make clones using the same allocator?

The current design uses static functions, and so there can be no stateful

...

...

allocator.

I thought a little about this, but have not come up with a solution. For example, how do we design an allocator that knows how to clone a class hierarchy?

Isn't this somewhat out of the scope of ptr_container?

Only the class itself knows how to clone itself because it

...

...

knows the exact type. OTOH, the classes in the class hieararchy doesn't know about the allocator.

We might try a new way to clone objects by replacing

virtual Base::clone() const = 0;

Isn't this what a copy constructor is for?

Types in object hierarchies usually don't expose their copy-constructor to avoid sliving.

But again, isn't this out of the scope? It's like the tail wagging the dog - ptr_container shouldn't really be concerned with object hierarchies or not. That's up to the user. Surely, copy construction must be the best default way to make a clone? The user can make their types noncopyable if needed.

Furthermore, and more importantly, you loose the concrete type of the objects by having a container of pointers to the base class. Virtual functions are the most elegant way to ask the object for a copy of the right type.

Isn't this about ptr_container? We can't really have the specific usage of OO hierarchies with containers of base pointers dictate the implementation of an otherwise general system. Introducing the idea of virtual methods and clone systems etc for ptr_container seems like feature creep. Isn't all that another system entirely? If the idea of clone_allocator was to give some customisation to creating a clone of a value in a ptr_container, then surely it should use the containers allocator, and it probably should use a copy constructor. In light of which, it seems that clone_allocator itself isn't really important or useful? If it really is there to try to do deep copies of object hierarchies, that seems to me to be part of an altogether different system and outside the scope of ptr_container. Otherwise, you should just use the containers allocator to make the copy.

[snip] This requires a public copy-constructor which we must not impose as a requirement, and even so it would not work because the concrete type is lost. (let's ignore that it is not exception-safe).

How will you make a clone without using a copy-constructor? Are you suggesting something like is_convertible<base *, T*> then switching to a virtual-method-based-scheme for clone?? I don't fully understand what you mean by "the concrete type is lost". Do you mean in this sort of case? struct Foo { vector<Base *> bases; Foo(const Foo &other) { ... /* how to clone bases? */ } }; If so, isn't that way out of the domain being addressed by ptr_container? But I think now I see more clearly what you mean. You want to help with just this case, is that right?

But this raises another question I have: would we want all the objects to be allocated from the same allocator type using rebind<>, or would we like to use two different allocators.

I would argue for rebinding. Allocators are supposed to be rebound, I don't think it is inelegant to do that. In any case, it still allows for the user to provide an explicit specialising for a given type if needed, so they don't lose control.

Point being, can you just pass the container's allocator to the

...

clone_allocator?

Or, pass in a rebound allocator to allocate_clone; it doesn't really matter.

I can, but I feel it is unanswered whether it should be the underlying container's allocator or simply the clone_allocator that somehow makes it's own allocations.

I don't understand why clone_allocator even needs to exist, let alone why it shouldn't use the containers allocator if it must exist. If the container allocator is good enough for values, it is good enough for clones. I think I see that you wanted a way to help with the Foo case above, but perhaps that really shouldn't be a part of ptr_container at all?

And, as stated above, we also need to make sure that implementers of clone() get the allocator.

It seems that you added clone_allocator to help with cloning of containers-of-base-pointers, but I can't see that working in general. Attempting to do so is really reaching from system library space into application space. If needed, the user can make their own clone() system, and use it to implement their own copy constructors, which would then be used by allocator::construct in a ptr_container to make a clone and clone_allocator then isn't even needed. Perhaps the idea of cloning can be grafted away from ptr_container and made into a seperate system. I can see why it's there, and the motivation, but if it was to work properly it would be of general use and then not a part of ptr_container anyway. Regards, Christian PS. Perhaps just porting ptr_container to use boost::container rather than std::container would solve Roberts problem?

PS. Perhaps just porting ptr_container to use boost::container rather than

...

std::container would solve Roberts problem?

No. He needs to "install" his own new/delete with the container.

Isn't this called a stateful allocator? Which boost::container supports? Regards, Christian.

Hi Thorsten, On Sat, Jun 27, 2009 at 11:48 AM, Thorsten Ottosen < thorsten.ottosen@dezide.com> wrote:

Christian Schladetsch skrev:

PS. Perhaps just porting ptr_container to use boost::container rather than

...

...

...

std::container would solve Roberts problem?

No. He needs to "install" his own new/delete with the container.

Isn't this called a stateful allocator? Which boost::container supports?

Hm. Is it? So construct() and destroy() would act as custom allocator/deleter?

No, the custom allocator would act as a custom allocator. Obviously I don't follow this. If you want to "install" custom new and delete into a container, isn't this best achieved by using a stateful allocator? If so, it seems the best want to support stateful allocators is to base ptr_container on boost::interprocess::containers (hopefully soon to be boost::containers) rather than std::containers. It seemed to me that Robert needed stateful allocators for ptr_containers. You said he didn't. But now you made clone_allocator stateful, but left the normal allocator stateless. I'm confused.

Anyway, let's say that all allocation is done with the supplied allocator via rebind<>. This is what you require to use ptr_container with monotonic allocators, right?

No. Monotonic works well with either stateless or stateful allocators. The problem was that I could not implement a sane clone_allocator for ptr_container, and I didn't know how anyone else would either. Now that you made clone_allocator stateful, I still can't pass it the containers allocator at constructon time. Why not make the parent allocator stateful, and pass it to a static clone method in a stateless clone_allocator? Doesn't that make more sense and solve more problems?

But how do you get objects into the ptr_container in the first place?

In a perfect world, I would pass a pointer to be added to a container of pointers: ptr_container<T> c; c.push_back(new T(..args..)); Then use value semantics of type T and not T * on the contained elements. But I realise that isn't how it works.

Do you intend users to write

container c; c.push_back( c.get_allocator().construct( c.get_allocator().allocate(sizeof(T)) );

No - why would they? That is what the container does internally. I am sure you are making a good point here, I just can't see it. All I need to make ptr_container work for me is to give the clone_allocator access to the parent container's allocator - preferably just by passing it to clone_allocator::clone rather than making it stateful. It makes sense to me that clones should be made using the parent container's own allocator, which it used to make the originals with in the first place. But then when you think that through, it seems that clone_allocator itself is a practically useless appendage. Unless you want clones to be made using a stateful sort-of allocator that does not model std::allocator, is different to the parent containers allocator, and you want to ensure that clones must be made using a different allocator to the parent container allocator. The obvious question then is: why you would want to do that, and how it is useful in any way? What would be useful is if ptr_container supported stateful v2 allocators based on boost::interprocess::containers, rather than stateless v1 allocators based on std::containers. Is this how you expect it to work for Robert and others?: struct my_stateful_custom_clone_allocator { ...state here, including some way to store a general allocator ... template <class U> U *clone(U const &X) { ...use the general allocator in this->state to make the clone of X... } }; ptr_container<T, my_stateful_custom_clone_allocator, my_custom_normal_allocator<T> > cont; my_custom_clone_allocator &alloc = cont.get_clone_allocator(); alloc.parent_allocator = cont.get_allocator(); alloc.... = ...; rather than: struct my_stateless_custom_clone_allocator { template <class U, class Alloc> U *clone(U const &X, Alloc &alloc) { // use alloc to make a clone of X } }; ptr_container<T, my_stateless_custom_clone_allocator, my_custom_normal_allocator<T> > cont; Having the users needing to dig into an otherwise private piece of machinery within the container itself - especially after it is created - seems really ugly. Isn't it better to do all the setup in the type and ctor for a ptr_container? I'm obivously missing something. Do you have any use-cases or examples that use a custom clone allocator with a ptr_container? Either stateless or stateful? I still think that the entire issue of cloning from base types is best put into a seperate, orthogonal system outside of ptr_container. However, I understand that you wanted to support it as a specific case for ptr_containers in the context of OO hierarchies. I also understand that you don't want to use copy-constructors to make a copy; there is no need to repeat either sentiment. And how do you pass constructor arguments to construct? Forwarding construction arguments is a general problem that is not limited to the std::allocator concept.

-Thorsten

Best, Christian

On Fri, Jun 26, 2009 at 9:36 PM, Zachary Turner <divisortheory@gmail.com>wrote:

It's a bit late in the thread, but I just wanted to mention that if your executable and DLL both link to the *shared* version of msvcrt (and the same shared version, e.g. mult-threaded debug, single-threaded release, etc) and not the static version of msvcrt, then you should have no problems deleting across dll boundaries. The problem occurs when two different modules have two different copies of the crt and one of those module tries to delete memory allocated by the other module.

The reason you often see windows api calls providing functions like NetApiFreeBuffer, etc is because they are implemented in a single dll (usually kernel32.dll) and there is only 1 copy of that dll on your system, that must work regardless of which version of the crt you link against.

Thanks for elaborating on these facts. I've indeed invested a lot of time researching this information. As you mention, there are a certain number of requirements that must be met in order for deleting across DLL boundaries to function, I prefer to choose the path that works in 100% of all cases, and that is to use factory functions and deleter methods. While this is not as clean, it is more fail safe. On a more philosophical note, I tend to treat deleting across DLL boundaries no different than sending a pointer across a network connection and calling 'delete' from a different computer. In my opinion, it is good general practice to maintain resources in a modular way. Not only does this have a greater chance of functioning, but it also makes the logic clearer and easier to follow. Again, thanks for reiterating these critical points.

Hi Thorsten, [snip]

able to copy e.g. ptr_vector<Base> is a key property of the library, and there is no way that is going to be removed from the library.

You can accept that or not. If you don't, we have nothing more to discuss. If you do, I would be happy to work with you and others to make the library even more useful, including finding someway of supporting that all memory is allocated from the same underlying allocator.

I have no problem with the library in general. I simply found it impossible to make a clone when using it. I've made my suggestion of passing the parent allocator to the clone_allocators static clone method. I'm happy to leave it at that. At the end of the day, I just change my local branch anyway ;) I'll think about a way of doing it that you may find preferable over passing the allocator to the clone_allocator. Regards, Christian

On Friday, June 26, 2009 6:35 AM Thorsten Ottosen wrote:

Christian Schladetsch skrev:

...

...

Types in object hierarchies usually don't expose their copy-constructor to avoid slicing.

But again, isn't this out of the scope? It's like the tail wagging the dog - ptr_container shouldn't really be concerned with object hierarchies or not. That's up to the user. [snip]

...

Furthermore, and more importantly, you lose the concrete type of the objects by having a container of pointers to the base class. Virtual functions are the most elegant way to ask the object for a copy of the right type.

Isn't this about ptr_container? We can't really have the specific usage of OO hierarchies with containers of base pointers dictate the implementation of an otherwise general system.

Please understand that pointer containers are written almost for the sole purpose of supporting OO programming better.

I know of use cases in which a similar container is used to hold homogeneous collections of objects by pointer, so cloning is irrelevant. Indeed, I recall that Rogue Wave had such containers available. Here's one: http://www2.roguewave.com/support/docs/hppdocs/tlsref/rwtptrvector.html. Can't cloning be handled using a smart pointer in a value-based container? That is, rather than make the pointer containers try to manage cloning, let them assume T is the most derived type rather than what might be a base type, and then supply a smart pointer for use with value-based containers to enable cloning. The smart pointer can call T::clone() or defer to a policy class to copy the T * it holds. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

Robert Dailey skrev:

On Fri, Jun 26, 2009 at 6:42 PM, Thorsten Ottosen < thorsten.ottosen@dezide.com> wrote:

...

Let me know if this allows you to solve your problem.

Thanks Thorsten,

I'm not familiar with how to use clone allocators in ptr_container yet, so I'll have to read through the documentation first to make sure I know how to use them. Given that, at what point will I need to be calling get_clone_allocator()? I figured I would just give ptr_vector my clone allocator and never need it back from the container. I expect my clone allocator to call the appropriate allocation method in its implementation details, which as I showed in an earlier example, will be a function located in the DLL itself.

In another thread you said:

class clone_allocator { public: clone_allocator() { dll_clone_object = GetProcAddress(); } MyClass* operator() ( MyClass const& node ) { return dll_clone_object( node ) }

private: MyClass* (*dll_clone_object)( MyClass const& ); };

Note that the above is pseudocode. The point is, it would allow me to abstract the DLL factory function for creating clones, thus giving me full control over which 'delete' and 'new' method is called (In this case, it would be the one from the DLL's memory manager).

so simply create a class like above, but implement the two functions described here: http://www.boost.org/doc/libs/1_39_0/libs/ptr_container/doc/reference.html#t... but make sure your my_clone_allocator has 1. non-static methods 2. is default constrictible 3. is swappable Then create a container (for example) like this: ptr_vector<T,my_clone_allocator> my_container; my_container.get_clone_allocator().set_allocate_new = ...; my_container.get_clone_allocator().set_deallocate = ...; or something. -Thorsten