Severe shared_ptr limitation WRT dynamically loaded libraries
Hello, I would like to raise what I think is a serious limitation of the boost shared_ptr implementation. I have thought a bit about solutions but don't have anything concrete to offer as yet but would be interested in seeing the thoughts of the boost library developers. I work in an architecture that uses dynamically loaded factory patterns quite a lot and factories can also be dynamically unloaded. This also makes heavy use of shared_ptrs to make memory management a breeze. The basic problem is that the boost implementation of the sp_counted_base relies on virtual functions to deallocate which are implemented by various templates in the detail namespace. If I have a dynamically loaded factory interface that returns a shared_ptr, the factory implementation causes the templates to generate the virtual function code within the factory library. Everything is fine as long as the library stays loaded but if the library is unloaded, any client code holding the shared pointer to an object generated by that factory now has an invalid virtual function table for the sp_counted_base it holds. So when the reference count goes to zero an access violation is generated. It is stated that the shared_ptr is to be as close to a raw pointer as possible which I very much agree with. However this behavior really sets it apart from raw pointers as it is certainly possible for a factory to return a dynamically allocated raw pointer to client code, unload the factory library and have the client code deallocate the raw pointer after the unload with no problem. To illustrate the problem in a windows environment, here is a simple example. Lets say I have a simple factory interface like so: class Factory { public: virtual shared_ptr<int> GetIntSharedPtr() = 0; virtual int * GetIntRawPtr() = 0; }; I implement a factory DLL to provide one of these factories that has an exported function GetFactory() like so: /// implementation of the Factory interface class TestFactory : public Factory { public: virtual shared_ptr<int> GetIntSharedPtr() { return shared_ptr<int>( new int(5) ); } virtual int * GetIntRawPtr() { return new int(0); } }; /// Library instance of the factory implementation TestFactory g_Factory; /// Exported function to return the factory for this library _declspec(dllexport) extern "C" Factory & GetFactory() { return g_Factory; } This is compiled into a library named "TestFactory.dll". Now I create a simple command line application that does the following: /// Typedefinition for for the GetFactory exported function typedef Factory & (_cdecl * GetFactory)(); int main() { // dynamically load the TestFactory library HMODULE hFactoryDll = ::LoadLibrary( "TestFactory.dll" ); // get the GetFactory() interface from the loaded library GetFactory pfnGetFactory = ::GetProcAddress( hFactoryDll, "GetFactory" ); // Acquire the factory object Factory & factory = pfnGetFactory(); // Call the factory interfaces to get dynamically allocated integers int * pRawInt = factory.GetIntRawPtr(); shared_ptr<int> spInt = factory.GetIntSharedPtr(); // everything is fine so far, now unload the factory library ::FreeLibrary( hFactoryDll ); // deallocating the raw pointer recieved from the factory works just fine delete pRawInt; // THIS WORKS! // However due to the reasons outlined in this post, releasing the shared_ptr causes // an access violation spInt.reset() // THIS CRASHES! } I would like to see a solution that would make this simple example and very reasonable use case work for shared_ptrs. Note that the argument that the client code cannot know what the factory is actually returning requires that the factory library be in memory so that appropriate subclass destructors can be called does not hold water here. This example uses simple integers. The root of the issue is not one of the virtual function table for the wrapped type getting corruped on library unload but rather the virtual function table for a member of the shared_ptr itself. I will put some thought toward this but am interested to see what boost developers might be able to come up with as well. J.D.
I would like to see a solution that would make this simple example and very reasonable use case work for shared_ptrs. Note that the argument that the client code cannot know what the factory is actually returning requires that the factory library be in memory so that appropriate subclass destructors can be called does not hold water here. This example uses simple integers. The root of the issue is not one of the virtual function table for the wrapped type getting corruped on library unload but rather the virtual function table for a member of the shared_ptr itself. I will put some thought toward this but am interested to see what boost developers might be able to come up with as well.
There is one aspect that you did not comment on, which is the allocation of memory. Whichever DLL allocates the memory must also deallocate it. If that DLL could also provide the shared_ptr, the problem disappears. Granted, it is easy to make a shared DLL provide the memory by simply using dynamic CRT linking, so the shared_ptr behavior is probably still undesirable. Arno -- Dr. Arno Schoedl · aschoedl@think-cell.com Technical Director think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany http://www.think-cell.com · phone +49-30-666473-10 · toll-free (US) +1-800-891-8091 Directors: Dr. Markus Hannebauer, Dr. Arno Schoedl · Amtsgericht Charlottenburg, HRB 85229
On Thu, Nov 20, 2008 at 3:25 AM, Arno Schödl <aschoedl@think-cell.com>wrote:
I would like to see a solution that would make this simple example and very reasonable use case work for shared_ptrs. Note that the argument that the client code cannot know what the factory is actually returning requires that the factory library be in memory so that appropriate subclass destructors can be called does not hold water here. This example uses simple integers. The root of the issue is not one of the virtual function table for the wrapped type getting corruped on library unload but rather the virtual function table for a member of the shared_ptr itself. I will put some thought toward this but am interested to see what boost developers might be able to come up with as well.
There is one aspect that you did not comment on, which is the allocation of memory. Whichever DLL allocates the memory must also deallocate it. If that DLL could also provide the shared_ptr, the problem disappears.
Granted, it is easy to make a shared DLL provide the memory by simply using dynamic CRT linking, so the shared_ptr behavior is probably still undesirable.
Arno
-- Dr. Arno Schoedl · aschoedl@think-cell.com Technical Director
think-cell Software GmbH · Invalidenstr. 34 · 10115 Berlin, Germany http://www.think-cell.com · phone +49-30-666473-10 · toll-free (US) +1-800-891-8091 Directors: Dr. Markus Hannebauer, Dr. Arno Schoedl · Amtsgericht Charlottenburg, HRB 85229
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Our architecture makes sure everything uses a shared CRT dll so actually as my example shows with the raw pointer you can allocate in one dll and deallocate in another executable. J.D.
On Wed, Nov 19, 2008 at 2:37 PM, J.D. Herron <jotadehace@gmail.com> wrote:
If I have a dynamically loaded factory interface that returns a shared_ptr, the factory implementation causes the templates to generate the virtual function code within the factory library. Everything is fine as long as the library stays loaded but if the library is unloaded, any client code holding the shared pointer to an object generated by that factory now has an invalid virtual function table for the sp_counted_base it holds. So when the reference count goes to zero an access violation is generated.
One solution is to "inject" the shared_ptr that would crash if a DLL is unloaded with another shared_ptr that keeps the DLL afloat. I've also done this with the boost::function objects that are passed to the client to be used as factories. Another strategy which doesn't necessarily work in the case of shared_ptr but works for other things (like std::type_infos obtained from a DLL) is to use weak_ptr to detect that the DLL is gone and accessing the object will crash the program, and throw an exception instead. Emil Dotchevski Reverge Studios, Inc. http://www.revergestudios.com/reblog/index.php?n=ReCode
On Thu, Nov 20, 2008 at 05:17:56AM -0800, Emil Dotchevski wrote:
On Wed, Nov 19, 2008 at 2:37 PM, J.D. Herron <jotadehace@gmail.com> wrote:
If I have a dynamically loaded factory interface that returns a shared_ptr, the factory implementation causes the templates to generate the virtual function code within the factory library. Everything is fine as long as the library stays loaded but if the library is unloaded, any client code holding the shared pointer to an object generated by that factory now has an invalid virtual function table for the sp_counted_base it holds. So when the reference count goes to zero an access violation is generated.
One solution is to "inject" the shared_ptr that would crash if a DLL is unloaded with another shared_ptr that keeps the DLL afloat. I've also done this with the boost::function objects that are passed to the client to be used as factories.
If you're talking workarounds, here's another one: why return shared_ptr at all? If it's a factory you're implementing (i.e. something that creates objects, and doesn't hold references to them), there's little reason to return one. Return a raw pointer, and build a shared_ptr from the return value. Except for horrible edge cases where between object creation and returning the pointer something prevents you from returning the pointer, you'll have the same results. Of course if you're not implementing a pure factory, but do retain pointers to the created object within the DLL, that won't work. Jens
On Thu, Nov 20, 2008 at 5:23 AM, Jens Finkhäuser <jens@unwesen.de> wrote:
On Thu, Nov 20, 2008 at 05:17:56AM -0800, Emil Dotchevski wrote:
On Wed, Nov 19, 2008 at 2:37 PM, J.D. Herron <jotadehace@gmail.com> wrote:
If I have a dynamically loaded factory interface that returns a shared_ptr, the factory implementation causes the templates to generate the virtual function code within the factory library. Everything is fine as long as the library stays loaded but if the library is unloaded, any client code holding the shared pointer to an object generated by that factory now has an invalid virtual function table for the sp_counted_base it holds. So when the reference count goes to zero an access violation is generated.
One solution is to "inject" the shared_ptr that would crash if a DLL is unloaded with another shared_ptr that keeps the DLL afloat. I've also done this with the boost::function objects that are passed to the client to be used as factories.
If you're talking workarounds, here's another one: why return shared_ptr at all? If it's a factory you're implementing (i.e. something that creates objects, and doesn't hold references to them), there's little reason to return one. Return a raw pointer, and build a shared_ptr from the return value.
Certainly, the most portable DLL interfaces would be limited to C only. My strategy is to expand the domain of things that work across DLL boundary just a bit over C, and include boost::shared_ptr and boost::function in it (that is, prevent silly problems like invalid vtbl pointers, and assume that everything else works.) Emil Dotchevski Reverge Studios, Inc. http://www.revergestudios.com/reblog/index.php?n=ReCode
Thank you Jens for this suggestion. Its a good one except for one limitation which forces all factories to dynamically allocate raw pointers that can be cleaned up with delete. I would like the flexibility of the factory to return shared_ptr so that it can use a NULL deleter to return non-dynamically allocated objects or use their own custom deleter to allow creation from stuff like pools. J.D. On Thu, Nov 20, 2008 at 6:23 AM, Jens Finkhäuser <jens@unwesen.de> wrote:
On Wed, Nov 19, 2008 at 2:37 PM, J.D. Herron <jotadehace@gmail.com> wrote:
If I have a dynamically loaded factory interface that returns a shared_ptr, the factory implementation causes the templates to generate
On Thu, Nov 20, 2008 at 05:17:56AM -0800, Emil Dotchevski wrote: the
virtual function code within the factory library. Everything is fine as long as the library stays loaded but if the library is unloaded, any client code holding the shared pointer to an object generated by that factory now has an invalid virtual function table for the sp_counted_base it holds. So when the reference count goes to zero an access violation is generated.
One solution is to "inject" the shared_ptr that would crash if a DLL is unloaded with another shared_ptr that keeps the DLL afloat. I've also done this with the boost::function objects that are passed to the client to be used as factories.
If you're talking workarounds, here's another one: why return shared_ptr at all? If it's a factory you're implementing (i.e. something that creates objects, and doesn't hold references to them), there's little reason to return one. Return a raw pointer, and build a shared_ptr from the return value. Except for horrible edge cases where between object creation and returning the pointer something prevents you from returning the pointer, you'll have the same results.
Of course if you're not implementing a pure factory, but do retain pointers to the created object within the DLL, that won't work.
Jens _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Thursday 20 November 2008 14:23:49 Jens Finkhäuser wrote:
If you're talking workarounds, here's another one: why return shared_ptr at all? If it's a factory you're implementing (i.e. something that creates objects, and doesn't hold references to them), there's little reason to return one. Return a raw pointer, and build a shared_ptr from the return value.
I would have said the same, but IMHO it should return std::auto_ptr instead. In cases where it returns a derived type, splitting into a public, nonvirtual function returning auto_ptr<BaseT> and a pure virtual one that can then support covariant return types is a suitable solution. Uli
Thanks for your suggestion. I've used this strategy for other things but it just seems like an awful lot of trouble for something I would like to just act like a raw pointer. J.D. On Thu, Nov 20, 2008 at 6:17 AM, Emil Dotchevski <emil@revergestudios.com>wrote:
If I have a dynamically loaded factory interface that returns a shared_ptr, the factory implementation causes the templates to generate
On Wed, Nov 19, 2008 at 2:37 PM, J.D. Herron <jotadehace@gmail.com> wrote: the
virtual function code within the factory library. Everything is fine as long as the library stays loaded but if the library is unloaded, any client code holding the shared pointer to an object generated by that factory now has an invalid virtual function table for the sp_counted_base it holds. So when the reference count goes to zero an access violation is generated.
One solution is to "inject" the shared_ptr that would crash if a DLL is unloaded with another shared_ptr that keeps the DLL afloat. I've also done this with the boost::function objects that are passed to the client to be used as factories.
Another strategy which doesn't necessarily work in the case of shared_ptr but works for other things (like std::type_infos obtained from a DLL) is to use weak_ptr to detect that the DLL is gone and accessing the object will crash the program, and throw an exception instead.
Emil Dotchevski Reverge Studios, Inc. http://www.revergestudios.com/reblog/index.php?n=ReCode _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Thu, Nov 20, 2008 at 8:16 AM, J.D. Herron <jotadehace@gmail.com> wrote:
Thanks for your suggestion. I've used this strategy for other things but it just seems like an awful lot of trouble for something I would like to just act like a raw pointer.
I think that this is the only viable strategy because for good reasons shared_ptr (including std::shared_ptr) leaves this type of things unspecified (not to mention DLLs are outside of the scope of the C++ standard anyway.) Also, the user is completely insulated from the "awful lot of trouble" you're talking about. Emil Dotchevski Reverge Studios, Inc. http://www.revergestudios.com/reblog/index.php?n=ReCode
When the shared_ptr is constructed pass in a custom deleter which will among other things increment and decrement the library reference count by calling LoadLibrary[Ex] and FreeLibrary. Note using GetModuleHandleEx might be better than load library for incrementing reference count as the library has already been loaded less needs to be done. -----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of J.D. Herron Sent: Wednesday, November 19, 2008 5:38 PM To: boost@lists.boost.org Subject: [boost] Severe shared_ptr limitation WRT dynamically loadedlibraries Hello, I would like to raise what I think is a serious limitation of the boost shared_ptr implementation. I have thought a bit about solutions but don't have anything concrete to offer as yet but would be interested in seeing the thoughts of the boost library developers. I work in an architecture that uses dynamically loaded factory patterns quite a lot and factories can also be dynamically unloaded. This also makes heavy use of shared_ptrs to make memory management a breeze. The basic problem is that the boost implementation of the sp_counted_base relies on virtual functions to deallocate which are implemented by various templates in the detail namespace. If I have a dynamically loaded factory interface that returns a shared_ptr, the factory implementation causes the templates to generate the virtual function code within the factory library. Everything is fine as long as the library stays loaded but if the library is unloaded, any client code holding the shared pointer to an object generated by that factory now has an invalid virtual function table for the sp_counted_base it holds. So when the reference count goes to zero an access violation is generated. It is stated that the shared_ptr is to be as close to a raw pointer as possible which I very much agree with. However this behavior really sets it apart from raw pointers as it is certainly possible for a factory to return a dynamically allocated raw pointer to client code, unload the factory library and have the client code deallocate the raw pointer after the unload with no problem. To illustrate the problem in a windows environment, here is a simple example. Lets say I have a simple factory interface like so: class Factory { public: virtual shared_ptr<int> GetIntSharedPtr() = 0; virtual int * GetIntRawPtr() = 0; }; I implement a factory DLL to provide one of these factories that has an exported function GetFactory() like so: /// implementation of the Factory interface class TestFactory : public Factory { public: virtual shared_ptr<int> GetIntSharedPtr() { return shared_ptr<int>( new int(5) ); } virtual int * GetIntRawPtr() { return new int(0); } }; /// Library instance of the factory implementation TestFactory g_Factory; /// Exported function to return the factory for this library _declspec(dllexport) extern "C" Factory & GetFactory() { return g_Factory; } This is compiled into a library named "TestFactory.dll". Now I create a simple command line application that does the following: /// Typedefinition for for the GetFactory exported function typedef Factory & (_cdecl * GetFactory)(); int main() { // dynamically load the TestFactory library HMODULE hFactoryDll = ::LoadLibrary( "TestFactory.dll" ); // get the GetFactory() interface from the loaded library GetFactory pfnGetFactory = ::GetProcAddress( hFactoryDll, "GetFactory" ); // Acquire the factory object Factory & factory = pfnGetFactory(); // Call the factory interfaces to get dynamically allocated integers int * pRawInt = factory.GetIntRawPtr(); shared_ptr<int> spInt = factory.GetIntSharedPtr(); // everything is fine so far, now unload the factory library ::FreeLibrary( hFactoryDll ); // deallocating the raw pointer recieved from the factory works just fine delete pRawInt; // THIS WORKS! // However due to the reasons outlined in this post, releasing the shared_ptr causes // an access violation spInt.reset() // THIS CRASHES! } I would like to see a solution that would make this simple example and very reasonable use case work for shared_ptrs. Note that the argument that the client code cannot know what the factory is actually returning requires that the factory library be in memory so that appropriate subclass destructors can be called does not hold water here. This example uses simple integers. The root of the issue is not one of the virtual function table for the wrapped type getting corruped on library unload but rather the virtual function table for a member of the shared_ptr itself. I will put some thought toward this but am interested to see what boost developers might be able to come up with as well. J.D. _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Thank you for another good suggestion Jarrad. This could certainly be done and would get around the problems with some other workarounds but it now forces all developers of factories to construct the shared_ptrs with special deleters. Granted this is really just a slight inconvenience but I'm also dealing with lots of legacy. This is certainly a viable work around but I'd still like to see a way that the shared_ptr implementation itself could handle this situation better. J.D. On Thu, Nov 20, 2008 at 6:47 AM, Jarrad Waterloo <jwaterloo@dynamicquest.com
wrote:
When the shared_ptr is constructed pass in a custom deleter which will among other things increment and decrement the library reference count by calling LoadLibrary[Ex] and FreeLibrary. Note using GetModuleHandleEx might be better than load library for incrementing reference count as the library has already been loaded less needs to be done.
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of J.D. Herron Sent: Wednesday, November 19, 2008 5:38 PM To: boost@lists.boost.org Subject: [boost] Severe shared_ptr limitation WRT dynamically loadedlibraries
Hello, I would like to raise what I think is a serious limitation of the boost shared_ptr implementation. I have thought a bit about solutions but don't have anything concrete to offer as yet but would be interested in seeing the thoughts of the boost library developers. I work in an architecture that uses dynamically loaded factory patterns quite a lot and factories can also be dynamically unloaded. This also makes heavy use of shared_ptrs to make memory management a breeze. The basic problem is that the boost implementation of the sp_counted_base relies on virtual functions to deallocate which are implemented by various templates in the detail namespace. If I have a dynamically loaded factory interface that returns a shared_ptr, the factory implementation causes the templates to generate the virtual function code within the factory library. Everything is fine as long as the library stays loaded but if the library is unloaded, any client code holding the shared pointer to an object generated by that factory now has an invalid virtual function table for the sp_counted_base it holds. So when the reference count goes to zero an access violation is generated. It is stated that the shared_ptr is to be as close to a raw pointer as possible which I very much agree with. However this behavior really sets it apart from raw pointers as it is certainly possible for a factory to return a dynamically allocated raw pointer to client code, unload the factory library and have the client code deallocate the raw pointer after the unload with no problem.
To illustrate the problem in a windows environment, here is a simple example. Lets say I have a simple factory interface like so:
class Factory { public: virtual shared_ptr<int> GetIntSharedPtr() = 0; virtual int * GetIntRawPtr() = 0; };
I implement a factory DLL to provide one of these factories that has an exported function GetFactory() like so:
/// implementation of the Factory interface class TestFactory : public Factory { public: virtual shared_ptr<int> GetIntSharedPtr() { return shared_ptr<int>( new int(5) ); }
virtual int * GetIntRawPtr() { return new int(0); } };
/// Library instance of the factory implementation TestFactory g_Factory;
/// Exported function to return the factory for this library _declspec(dllexport) extern "C" Factory & GetFactory() { return g_Factory; }
This is compiled into a library named "TestFactory.dll". Now I create a simple command line application that does the following:
/// Typedefinition for for the GetFactory exported function typedef Factory & (_cdecl * GetFactory)();
int main() { // dynamically load the TestFactory library HMODULE hFactoryDll = ::LoadLibrary( "TestFactory.dll" );
// get the GetFactory() interface from the loaded library GetFactory pfnGetFactory = ::GetProcAddress( hFactoryDll, "GetFactory" );
// Acquire the factory object Factory & factory = pfnGetFactory();
// Call the factory interfaces to get dynamically allocated integers int * pRawInt = factory.GetIntRawPtr();
shared_ptr<int> spInt = factory.GetIntSharedPtr();
// everything is fine so far, now unload the factory library ::FreeLibrary( hFactoryDll );
// deallocating the raw pointer recieved from the factory works just fine delete pRawInt; // THIS WORKS!
// However due to the reasons outlined in this post, releasing the shared_ptr causes // an access violation spInt.reset() // THIS CRASHES! }
I would like to see a solution that would make this simple example and very reasonable use case work for shared_ptrs. Note that the argument that the client code cannot know what the factory is actually returning requires that the factory library be in memory so that appropriate subclass destructors can be called does not hold water here. This example uses simple integers. The root of the issue is not one of the virtual function table for the wrapped type getting corruped on library unload but rather the virtual function table for a member of the shared_ptr itself. I will put some thought toward this but am interested to see what boost developers might be able to come up with as well.
J.D. _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
participants (6)
-
Arno Schödl -
Emil Dotchevski -
J.D. Herron -
Jarrad Waterloo -
Jens Finkhäuser -
Ulrich Eckhardt