[boost::serialization] - serialization + shared_ptrs
Hi there, After fixing header include orders for the umpteenth time to ensure boost/serialization/shared_ptr.hpp is included before shared_ptr.hpp so it can do it's #define private public hack I wondered if this was any closer to an acceptable resolution? I know this is brought up periodcally but it really needs to be sorted out in someway, currently I have just hacked my local boost source to make shared_ptr's private declarations public to avoid the issue but obviously this is far from ideal. Thanks Martin
Hi,
After fixing header include orders for the umpteenth time to ensure boost/serialization/shared_ptr.hpp is included before shared_ptr.hpp so it can do it's #define private public hack I wondered if this was any closer to an acceptable resolution? I know this is brought up periodcally but it really needs to be sorted out in someway
I fully agree. Not only does the private-public hack bother me, but also the fact that a special export macro BOOST_SHARED_POINTER_EXPORT is needed to export classes used with shared_ptr. It's especially cumbersome when shared_ptr is used polymorphically, i.e. we use shared_ptr<Base> that actually points to an instance of class Derived. Also, if I'm not mistaken, the current scheme does not support weak_ptr. Since shared_ptr is such an important and often used class (at least at my company; we use it almost exclusively instead of raw pointers) , there should be proper, built-in support in boost::serialization for it. No extra macro or intrusive hack to shared_ptr should be required in order to be able to use it. As was discussed before, a good way to achieve this is to use a pointer-to-shared_ptr map in the input archive when loading archives that contain shared_ptrs. This is actually the approach that we've hacked into the version of boost::serialization here at work. We've been using it for a couple of months now like this and have not experienced any problems with it. And it also works for serializing weak_ptr. However, I would really appreciate an 'official' solution to shared_ptr (and weak_ptr) serialization. Best Regards, Martin TAB Austria Haiderstraße 40 4052 Ansfelden Austria Phone: +43 7229 78040-218 Fax: +43 7229 78040-209 E-mail: martin.ecker@tab.at http://www.tab.at
martin.ecker@tab.at writes:
shared_ptr is such an important and often used class (at least at my company; we use it almost exclusively instead of raw pointers)
How is that policy working out for you? Some people I know are convinced that going that way leads to bad, slow code. Are your environments resource-constrained at all? -- Dave Abrahams Boost Consulting www.boost-consulting.com
How is that policy working out for you? Some people I know are convinced that going that way leads to bad, slow code. Are your environments resource-constrained at all?
Just though I'd mention my experiences here as in our last project I heavily used shared pointers (literally only wrote delete once or twice in 6-12 months) and had absolutely no problems. In the industry I'm in (games) performance is taken extremely seriously and there is much fud about what is good / bad for performance. I spent many many hours with vtune profiling every aspect of the game and at no point found any place where shared pointers were the problem at all. This was a PC product, so pretty much unconstrained but after this I would definately look at using it in next generation console environments (embedded systems really) with a suitable allocator to prevent fragmentation. cheers Martin
Hello, David Abrahams wrote:
How is that policy working out for you? Some people I know are convinced that going that way leads to bad, slow code. Are your environments resource-constrained at all?
Martin Slater wrote:
In the industry I'm in (games) performance is taken extremely seriously and there is much fud about what is good / bad for performance. I spent many many hours with vtune profiling every aspect of the game and at no point found any place
where shared pointers were the problem at all.
We're also in the games industry and we basically develop for the PC (for arcade machines, though), and similar to Martin we are not experiencing any problems (performance, size or otherwise) with shared_ptr. But then again, we're not developing for a PS2 or similar consoles. Regards, Martin TAB Austria Haiderstraße 40 4052 Ansfelden Austria Phone: +43 7229 78040-218 Fax: +43 7229 78040-209 E-mail: martin.ecker@tab.at http://www.tab.at
martin.ecker@tab.at wrote:
also the fact that a special export macro BOOST_SHARED_POINTER_EXPORT is needed to export classes used with shared_ptr. It's especially cumbersome when shared_ptr is used polymorphically, i.e. we use shared_ptr<Base> that actually points to an instance of class Derived.
Note that this should be required ONLY when shared_ptr is used polymorphically.
there should be proper, built-in support in boost::serialization for it.
agreed.
No extra macro or intrusive hack to shared_ptr should be required in order to be able to use it.
The macro described above is required to instantiate code not explicitly referred to. This occurs with the serialization of derived pointers through a polymorphic base class. There are two ways to do this: explicit registration and export. Each one has its drawbacks. The only alternative that has been suggested is to use the type_info name as a class identifier. I didn't do this as it would inhibit portability of archives across compilers and namespaces.
As was discussed before, a good way to achieve this is to use a pointer-to-shared_ptr map in the input archive when loading archives that contain shared_ptrs. This is actually the approach that we've hacked into the version of boost::serialization here at work. We've been using it for a couple of months now like this and have not experienced any problems with it.
Gee, wouldn't it have been easier just to add a friend to the share_ptr?
And it also works for serializing weak_ptr.
However, I would really appreciate an 'official' solution to shared_ptr (and weak_ptr) serialization.
Me too - but I can't see how this is going to happen Robert Ramey
Robert Ramey <ramey <at> rrsd.com> writes:
martin.ecker <at> tab.at wrote:
also the fact that a special export macro BOOST_SHARED_POINTER_EXPORT is needed to export classes used with shared_ptr. It's especially cumbersome when shared_ptr is used polymorphically, i.e. we use shared_ptr<Base> that actually points to an instance of class Derived.
Requiring extra registration macros for shared_ptr<T>, beyond the ones needed to serialize T*, to me indicates a deeper flaw with how Boost.Serialization handles shared_ptr's.
As was discussed before, a good way to achieve this is to use a pointer-to-shared_ptr map in the input archive when loading archives that contain shared_ptrs. This is actually the approach that we've hacked into the version of boost::serialization here at work. We've been using it for a couple of months now like this and have not experienced any problems with it.
Gee, wouldn't it have been easier just to add a friend to the share_ptr?
And it also works for serializing weak_ptr.
However, I would really appreciate an 'official' solution to shared_ptr (and weak_ptr) serialization.
Me too - but I can't see how this is going to happen
Robert Ramey
Why can't it happen? There was a thread on this a while back, presenting the same solution as above. Is there a particular reason why that approach wasn't taken? Was it ever considered? I've implemented it myself, both in my own serialization framework (http: //tinyurl.com/3nhmc), and on top of boost's, and it's pretty simple. 100% external serialization, and it would work with just about any reference-counted smart pointer. No extra registration macros needed either. The way it's done now just seems very heavy handed to me, and doing it that way for every smart pointer that might come along, eg in std::tr1, is not even feasible. Since its possible to serialize a shared_ptr, or any reasonable smart pointer for that matter, from its corresponding public interface, why not do it? /Jarl PS Thanks for an otherwise great library! shared_ptr's are my only gripe :)
Jarl Lindrud wrote:
Robert Ramey <ramey <at> rrsd.com> writes:
Requiring extra registration macros for shared_ptr<T>, beyond the ones needed to serialize T*, to me indicates a deeper flaw with how Boost.Serialization handles shared_ptr's.
Hmmm - how do you know it doesn't indicate a flaw in TR1::shared_pointer that doesn't specify an interface rich enough to support serialization as all the other standard library components do?
As was discussed before, a good way to achieve this is to use a pointer-to-shared_ptr map in the input archive when loading archives that contain shared_ptrs. This is actually the approach that we've hacked into the version of boost::serialization here at work. We've been using it for a couple of months now like this and have not experienced any problems with it.
Of course, you'll have to do this again if you want to use the next version of the serialization library - but that's your choice. If I had been faced with this problem and needed an immediate solution I would have made a derivation of the included archives along the lines of the following class text_iarchive_with_special_shared_ptr_handling : public text_iarchive { std::map<void *, ?> shared_ptr_map; }; Now you have a map when you need it and everything is about done. Not need to meddle in the serialization library implemenation.
Why can't it happen?
a) The author of boost shared pointer doesn't want to include serialization implementation in his implementation of boost::shared_ptr b) I don't want to add code to the implementation of serialization to handle one particular data type. This strikes me as fragile, open ended and contrary to the whole idea of separating the serialization of the serialization library from the types it is meant to serialize. After implementing serialization for all the standard library types without major problem, we now want to make a special case for one particular type? Where does that lead us? What about the other types in TR1? Has anyone considered whether or not they expose enough of their state to permit serialization independent of their implemenation? The problem I'm concerned about is adding stuff into the serialization library just to handle one particular type. That creates a sort of open ended maintainence issue. I would hope someone might suggest some elegant edition to the interface that would be guarenteed to resolve
There was a thread on this a while back, presenting the same solution as above. Is there a particular reason why that approach wasn't taken? Was it ever considered?
This was in response to my request for access to the implementation of smart_ptr. My request wasn't really even considered. But the real answer is that although I understand the problem, I havn't figured out a general solution and I haven't heard one from anyone either.
I've implemented it myself, both in my own serialization framework (http: //tinyurl.com/3nhmc), and on top of boost's, and it's pretty simple. 100% external serialization, and it would work with just about any reference-counted smart pointer. No extra registration macros needed either.
Congratulations - maybe you want to upload to the file section
The way it's done now just seems very heavy handed to me,
no argument there
and doing it that way for every smart pointer that might come along, eg in std::tr1, is not even feasible. Since its possible to serialize a shared_ptr, or any reasonable smart pointer for that matter, from its corresponding public interface
I really can't know that one solution is going to work for every "reasonable" smart pointer. and even so it would require adding every new type that came along into the serialization library. This would defeat the original purpose of factoring out this code into a "library"
shared_ptr's are my only gripe :)
And since you've got your own copy fixed to taste, you must be a very happy man. For what its worth, my thoughts along these lines are something like: calss helper_base { virtual ~helper_base() = 0; } class text_iarchive ... std::list<helpbase *> helper_list; text_iarchive(){ for(it = help_list.begin();it != helper_list.end();++it) delete *it; } }; template<class T> shared_ptr_helper : public helper_base { std::map<void *> ptr_map; }; template<class Archive, class T> void serialize(Archive &ar, boost::shared_ptr<T> &t){ help_base * hbp = ar.find_helper(shared_ptr_helper<T>) if(NULL == hbp){ hbp = new share_ptr_helper<T>; ar.add(hbp); } // we now have the address of a map which will be deleted // when archive is deleted. // use map as necessary hbp->ptr_map ... ... } This would handle situatations like shared_ptr. I don't know if it would be general enough to handle them all. Its easy to design a class that is not serializable even in principle. I don't know about the other boost types addressed in tr1. Anyway, its only a suggestion. I'm still open to suggestions that address all the conflicting concerns raised so far. It seems that relatively few boost types have had serialization added. I presume there is a lack of usage of the combination of serializaton and these types. We did get boost::varient implemented. Ironically, this is implemented through only its public interface - at the cost of an extra copy even though its not part of any standard. Personally, I would have preferred to elminate the copy have have serialization an implementation feature of boost::varient but I didn't need any more aggravation. The latest updates to shared_pointer checked into CVS about 10 days ago break current serialzation of shared pointer. In no way will I have time to fix this before 15 April. So I would expect we'll have to deprecate shared pointer serialization for 1.33 . Current users of this combination will have to wait until Peter Dimov's "proper" implementation in 1.34. Robert Ramey
Hi, On Thu, Apr 07, 2005 at 10:13:35PM -0700, Robert Ramey wrote:
The latest updates to shared_pointer checked into CVS about 10 days ago break current serialzation of shared pointer. In no way will I have time to fix this before 15 April. So I would expect we'll have to deprecate shared pointer serialization for 1.33 . Current users of this combination will have to wait until Peter Dimov's "proper" implementation in 1.34.
I did not follow the discussions about lock free implementations for shared_ptr closely. But I saw that they make use of inline assembler whence there cannot be a lock free implementation that is portable across all platforms. I therefore gather that the previous implementation is still available as a fall-back for platforms without a lock-free implementation, isn't it? Can the old implementation be made available in 1.33 by means of a preprocessor switch? (Something like BOOST_SHARED_PTR_USE_LEGACY_CODE.) That would allow users of the serialization library to update to 1.33. The old implementation could still be declared deprecated on all platforms where a lock free implementation is available. I don't doubt that Peter Dimov tested the lock free shared_ptr implementations thoroughly. But it is my understanding that this is an area where subtle bugs lurk all over the place whence it is still possible that not anticipated problems arise after the release of 1.33. On the other hand, the old implementation was stable and has been used in many, many applications without problems. I imagine that many users will feel more comfortable about switching to 1.33 if the old implementation is still available as a safety net. Once the lock free implementations turned out to be stable and there is a solution for the serialization of shared_ptr the preprocessor switch I proposed above can be removed in a later release. Regards Christoph -- http://www.informatik.tu-darmstadt.de/TI/Mitarbeiter/cludwig.html LiDIA: http://www.informatik.tu-darmstadt.de/TI/LiDIA/Welcome.html
Christoph Ludwig wrote:
I did not follow the discussions about lock free implementations for shared_ptr closely. But I saw that they make use of inline assembler whence there cannot be a lock free implementation that is portable across all platforms. I therefore gather that the previous implementation is still available as a fall-back for platforms without a lock-free implementation, isn't it?
Can the old implementation be made available in 1.33 by means of a preprocessor switch?
On Windows, the old spinlock-based implementation is no longer available; the new implementation is better in every respect. On other platforms the old pthread_mutex implementation can be forced with BOOST_SP_USE_PTHREADS. In both cases a single-threaded version can be forced with BOOST_SP_DISABLE_THREADS.
Robert Ramey wrote:
a) The author of boost shared pointer doesn't want to include serialization implementation in his implementation of boost::shared_ptr
The author of boost::shared_ptr wants to, but it isn't straightforward and will take time; and you aren't helping me much, I might add. [...]
This was in response to my request for access to the implementation of smart_ptr. My request wasn't really even considered.
[...]
The latest updates to shared_pointer checked into CVS about 10 days ago break current serialzation of shared pointer. In no way will I have time to fix this before 15 April.
You can see now why I don't consider "access to the implementation" a viable approach. It is not reasonable to cease any further work on boost::shared_ptr because the serialization library depends on a specific implementation. Your other suggestion was "add a serialize member to shared_ptr and just make it work." This is easier said than done. The serialization library is fairly complex and figuring out the best way to extend it non-intrusively with the necessary support requires more time than I have at the moment.
Peter Dimov wrote:
Robert Ramey wrote:
a) The author of boost shared pointer doesn't want to include serialization implementation in his implementation of boost::shared_ptr
The author of boost::shared_ptr wants to, but it isn't straightforward and will take time; and you aren't helping me much, I might add.
Hmmm I understood that you didn't want to include serialization inside the implementation of shared_ptr but had insisted on and totally non-intrusive one which depended only on public interface of shared_ptr. That's what I was refering to. And I'm not unsympathetic to this point of view. In fact its exactly the reason I want to avoid including code to implement one specific type (shared_ptr) inside the serialization implementation. Shifting the problem from one library to a different libray may be a solution for one developer - but its not a solution for boost as a whole. So far the only concrete constructive suggestion I've seen is the one I just posted.
The latest updates to shared_pointer checked into CVS about 10 days ago break current serialzation of shared pointer. In no way will I have time to fix this before 15 April.
You can see now why I don't consider "access to the implementation" a viable approach.
I never failed to see your point - I just never really saw an attractive resolution. Lacking such a resolution - I've "made it work" which is more than anyone else has done.
It is not reasonable to cease any further work on boost::shared_ptr because the serialization library depends on a specific implementation.
Maybe, but it will leave some developers in the lurch and the issue has been out there for over a year.
Your other suggestion was "add a serialize member to shared_ptr and just make it work." This is easier said than done. The serialization library is fairly complex and figuring out the best way to extend it non-intrusively with the necessary support requires more time than I have at the moment.
Welcome to the club. Robert Ramey
Robert Ramey wrote:
Peter Dimov wrote:
The author of boost::shared_ptr wants to, but it isn't straightforward and will take time; and you aren't helping me much, I might add.
Hmmm I understood that you didn't want to include serialization inside the implementation of shared_ptr but had insisted on and totally non-intrusive one which depended only on public interface of shared_ptr. That's what I was refering to.
Yes, I still want to use only the public interface of shared_ptr in its serialization. This will allow people to use the same serialization algorithm with tr1::shared_ptr.
Your other suggestion was "add a serialize member to shared_ptr and just make it work." This is easier said than done. The serialization library is fairly complex and figuring out the best way to extend it non-intrusively with the necessary support requires more time than I have at the moment.
Welcome to the club.
Good point. (apologies for quoting out of order)
I never failed to see your point - I just never really saw an attractive resolution. Lacking such a resolution - I've "made it work" which is more than anyone else has done.
I've attached my current draft version. The top part is the shared_ptr serialization support, the rest (after namespace boost is closed) is a simple test. It uses the ordinary BOOST_CLASS_EXPORT_GUID mechanism for class registration, but serialization via a base class requires an additional step: register_conversion<base, derived>(); at the top of main.cpp. The main thing that makes this a draft and not a production version are the two global std::map objects at the top, sp_save_map and sp_load_map. These need to be injected into the archive. The register_conversion step cannot be eliminated without adding specific shared_ptr support to the serialization library; its void_cast mechanism cannot be used to upcast a shared_ptr to shared_ptr. When I wrote the code I intended to provide a mechanism to inject the pointer maps into the archive types via inheritance. Then I ran out of time. So, basically, the only thing that remains to be done is a mechanism to inject sp_load_map into an iarchive, and sp_save_map into an oarchive. If you have (or anyone else has) an easy solution to this problem, that would be great.
I've found an implementation of shared pointer serialization that martin ecker sent me some time ago. It seems to me that it is simple transparent and would satisfy all requirements. My only concern is that helper class concept attachement to the archive needs to be generalized but this is doable. Robert Ramey begin 666 shared_ptr.hpp M(VEF;F1E9B!"3T]35%]315))04Q)6D%424].7U-(05)%1%]05%)?2%!0#0HC M9&5F:6YE($)/3U-47U-%4DE!3$E:051)3TY?4TA!4D5$7U!44E](4% -"@T* M+R\@35,@8V]M<&%T:6)L92!C;VUP:6QE<G,@<W5P<&]R=" C<')A9VUA(&]N M8V4-"B-I9B!D969I;F5D*%]-4T-?5D52*2 F)B H7TU30U]615(@/CT@,3 R M,"D-"B,@<')A9VUA(&]N8V4-"B-E;F1I9@T*#0HO+R\O+R\O+R\Q+R\O+R\O M+R\O,B\O+R\O+R\O+S,O+R\O+R\O+R\T+R\O+R\O+R\O-2\O+R\O+R\O+S8O M+R\O+R\O+R\W+R\O+R\O+R\O. T*+R\@<VAA<F5D7W!T<BYH<' Z('-E<FEA M;&EZ871I;VX@9F]R(&)O;W-T('-H87)E9"!P;VEN=&5R#0H-"B\O("A#*2!# M;W!Y<FEG:'0@,C P-"!2;V)E<G0@4F%M97D@86YD($UA<G1I;B!%8VME<@T* M+R\@57-E+"!M;V1I9FEC871I;VX@86YD(&1I<W1R:6)U=&EO;B!I<R!S=6)J M96-T('1O('1H92!";V]S="!3;V9T=V%R90T*+R\@3&EC96YS92P@5F5R<VEO M;B Q+C N("A3964@86-C;VUP86YY:6YG(&9I;&4@3$E#14Y315\Q7S N='AT M(&]R(&-O<'D@870-"B\O(&AT=' Z+R]W=W<N8F]O<W0N;W)G+TQ)0T5.4T5? M,5\P+G1X="D-"@T*+R\@(%-E92!H='1P.B\O=W=W+F)O;W-T+F]R9R!F;W(@ M=7!D871E<RP@9&]C=6UE;G1A=&EO;BP@86YD(')E=FES:6]N(&AI<W1O<GDN M#0H-"B-I;F-L=61E(#QB;V]S="]S:&%R961?<'1R+FAP<#X-"B-I;F-L=61E M(#QB;V]S="]W96%K7W!T<BYH<' ^#0HC:6YC;'5D92 \8F]O<W0O<V5R:6%L M:7IA=&EO;B]S<&QI=%]F<F5E+FAP<#X-"B-I;F-L=61E(#QB;V]S="]S97)I M86QI>F%T:6]N+VYV<"YH<' ^#0H-"FYA;65S<&%C92!B;V]S="![#0IN86UE M<W!A8V4@<V5R:6%L:7IA=&EO;GL-"@T*=&5M<&QA=&4\8VQA<W,@07)C:&EV M92P@8VQA<W,@5#X-"FEN;&EN92!V;VED('-A=F4H#0H@(" @07)C:&EV92 F M(&%R+ T*(" @(&-O;G-T(&)O;W-T.CIS:&%R961?<'1R/%0^("9T+ T*(" @ M(&-O;G-T('5N<VEG;F5D(&EN=" O*B!F:6QE7W9E<G-I;VX@*B\-"BE[#0H) M87(@/#P@8F]O<W0Z.G-E<FEA;&EZ871I;VXZ.FUA:V5?;G9P*")P>"(L('0N M9V5T*"DI.PT*?0T*#0IT96UP;&%T93QC;&%S<R!!<F-H:79E+"!C;&%S<R!4 M/@T*:6YL:6YE('9O:60@;&]A9"@-"B @("!!<F-H:79E("8@87(L#0H@(" @ M8F]O<W0Z.G-H87)E9%]P='(\5#X@)G0L#0H@(" @8V]N<W0@=6YS:6=N960@ M:6YT("\J(&9I;&5?=F5R<VEO;B J+PT**7L-"@E4*B!R.PT*"6%R(#X^(&)O M;W-T.CIS97)I86QI>F%T:6]N.CIM86ME7VYV<"@B<'@B+"!R*3L-"@ET(#T@ M87(N9V5T7W-H87)E9%]P=')?<F5G:7-T<GDH*2YL;V]K=7 H<BD[#0H):68@ M*"%T*0T*"7L-"@D)="YR97-E="AR*3L-"@D)87(N9V5T7W-H87)E9%]P=')? M<F5G:7-T<GDH*2YR96=I<W1E<E]S:&%R961?<'1R*'0I.PT*"7T-"GT-"@T* M=&5M<&QA=&4\8VQA<W,@07)C:&EV92P@8VQA<W,@5#X-"FEN;&EN92!V;VED M('-E<FEA;&EZ92@-"B @("!!<F-H:79E("8@87(L#0H@(" @8F]O<W0Z.G-H M87)E9%]P='(\5#X@)G0L#0H@(" @8V]N<W0@=6YS:6=N960@:6YT(&9I;&5? M=F5R<VEO;@T**7L-"B @("!B;V]S=#HZ<V5R:6%L:7IA=&EO;CHZ<W!L:71? M9G)E92AA<BP@="P@9FEL95]V97)S:6]N*3L-"GT-"@T*=&5M<&QA=&4\8VQA M<W,@07)C:&EV92P@8VQA<W,@5#X-"FEN;&EN92!V;VED('-A=F4H#0H@(" @ M07)C:&EV92 F(&%R+ T*(" @(&-O;G-T(&)O;W-T.CIW96%K7W!T<CQ4/B F M="P-"B @("!C;VYS="!U;G-I9VYE9"!I;G0@+RH@9FEL95]V97)S:6]N("HO M#0HI>PT*"6%R(#P\(&)O;W-T.CIS97)I86QI>F%T:6]N.CIM86ME7VYV<"@B M<'@B+"!T+FQO8VLH*2YG970H*2D[#0I]#0H-"G1E;7!L871E/&-L87-S($%R M8VAI=F4L(&-L87-S(%0^#0II;FQI;F4@=F]I9"!L;V%D* T*(" @($%R8VAI M=F4@)B!A<BP-"B @("!B;V]S=#HZ=V5A:U]P='(\5#X@)G0L#0H@(" @8V]N M<W0@=6YS:6=N960@:6YT("\J(&9I;&5?=F5R<VEO;B J+PT**7L-"@E4*B!R M.PT*"6%R(#X^(&)O;W-T.CIS97)I86QI>F%T:6]N.CIM86ME7VYV<"@B<'@B M+"!R*3L-"@EB;V]S=#HZ<VAA<F5D7W!T<CQ4/B!S<" ](&%R+F=E=%]S:&%R M961?<'1R7W)E9VES=')Y*"DN;&]O:W5P*'(I.PT*"6EF("@A<W I#0H)>PT* M"0ES<"YR97-E="AR*3L-"@D)87(N9V5T7W-H87)E9%]P=')?<F5G:7-T<GDH M*2YR96=I<W1E<E]S:&%R961?<'1R*'-P*3L-"@E]#0H-"@DO+R!A<W-I9VX@ M=&AE('-H87)E9%]P='(@=&\@=&AE('=E86M?<'1R#0H)=" ]('-P.PT*?0T* M#0IT96UP;&%T93QC;&%S<R!!<F-H:79E+"!C;&%S<R!4/@T*:6YL:6YE('9O M:60@<V5R:6%L:7IE* T*(" @($%R8VAI=F4@)B!A<BP-"B @("!B;V]S=#HZ M=V5A:U]P='(\5#X@)G0L#0H@(" @8V]N<W0@=6YS:6=N960@:6YT(&9I;&5? M=F5R<VEO;@T**7L-"B @("!B;V]S=#HZ<V5R:6%L:7IA=&EO;CHZ<W!L:71? M9G)E92AA<BP@="P@9FEL95]V97)S:6]N*3L-"GT-"@T*?2 O+R!N86UE<W!A M8V4@<V5R:6%L:7IA=&EO;@T*?2 O+R!N86UE<W!A8V4@8F]O<W0-"@T*(V5N %9&EF#0H` ` end
Robert Ramey wrote:
I've found an implementation of shared pointer serialization that martin ecker sent me some time ago. It seems to me that it is simple transparent and would satisfy all requirements. My only concern is that helper class concept attachement to the archive needs to be generalized but this is doable.
Robert Ramey
begin 666 shared_ptr.hpp M(VEF;F1E9B!"3T]35%]315))04Q)6D%424].7U-(05)%1%]05%)?2%!0#0HC M9&5F:6YE($)/3U-47U-%4DE!3$E:051)3TY?4TA!4D5$7U!44E](4% -"@T* M+R\@35,@8V]M<&%T:6)L92!C;VUP:6QE<G,@<W5P<&]R=" C<')A9VUA(&]N
Robert, Your attachment doesn't seem to have made it through to my mail client properly, any chance you could try again? thanks Martin -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.308 / Virus Database: 266.9.5 - Release Date: 7/04/2005
Robert Ramey wrote:
I've found an implementation of shared pointer serialization that martin ecker sent me some time ago. It seems to me that it is simple transparent and would satisfy all requirements.
Depends on whether your requirements include serializations of shared_ptr<X> for different X'es sharing ownership. template<class Archive, class T> inline void load( Archive & ar, boost::shared_ptr<T> &t, const unsigned int /* file_version */ ){ T* r; ar >> boost::serialization::make_nvp("px", r); t = ar.get_shared_ptr_registry().lookup(r); lookup(r) probably returns a shared_ptr<T> for the assignment to work... This probably means that there are separate map< T*, shared_ptr<T> > registries for each T. if (!t) { t.reset(r); ar.get_shared_ptr_registry().register_shared_ptr(t); } } This approach covers most of the real-world cases, so it's definitely better than the status quo. Go for it.
Robert Ramey <ramey <at> rrsd.com> writes:
Jarl Lindrud wrote:
Robert Ramey <ramey <at> rrsd.com> writes:
Requiring extra registration macros for shared_ptr<T>, beyond the ones needed to serialize T*, to me indicates a deeper flaw with how Boost.Serialization handles shared_ptr's.
Hmmm - how do you know it doesn't indicate a flaw in TR1::shared_pointer that doesn't specify an interface rich enough to support serialization as all the other standard library components do?
shared_ptr's, be they in boost or tr1, _do_ specify an interface rich enough to serialize through. If you can serialize straight pointers, then you can serialize smart pointers, as long as there is a mechanism in the archive to store user-defined objects during the duration of the serialization. In fact, if you have a map (void *, typeinfo *) -> void* in the archive, exposed to the serialization engine, you could then for a given pointer associate various diffent smart pointer objects, not just shared_ptr. The serialization code for a reference-counted smart pointer could be hidden behind a macro BOOST_SERIALIZE_REFCOUNTED_SMART_POINTER( smart_ptr ) and then it becomes trivial to define serialization for any smart pointer that eg supports operator= and get() (thats what i meant by "reasonable"): BOOST_SERIALIZE_REFCOUNTED_SMART_POINTER( tr1::shared_ptr ) BOOST_SERIALIZE_REFCOUNTED_SMART_POINTER( boost::shared_ptr ) BOOST_SERIALIZE_REFCOUNTED_SMART_POINTER( my_own_special_smart_ptr ) So its not a question of adding code to Boost.Serialization specifically for boost::shared_ptr, its a question of adding code for reference-counted smart pointers in general. Once thats done, serialization of just about any smart pointer can be implemented external to Boost.Serialization, through the macros. In fact, its just a question of adding the map described above to all the archives. Then anyone can serialize smart pointers externally to boost. serialization, by using the map as they please. And you can use the map for any other purposes that might arise, its not intrinsically linked to smart pointers.
Why can't it happen?
a) The author of boost shared pointer doesn't want to include serialization implementation in his implementation of boost::shared_ptr
And IMO he shouldnt, not inside the class boost::shared_ptr<> anyway.
b) I don't want to add code to the implementation of serialization to handle one particular data type. This strikes me as fragile, open ended and contrary to the whole idea of separating the serialization of the serialization library from the types it is meant to serialize. After implementing serialization for all the standard library types without major problem, we now want to make a special case for one particular type? Where does that lead us? What about the other types in TR1? Has anyone considered whether or not they expose enough of their state to permit serialization independent of their implemenation?
The problem I'm concerned about is adding stuff into the serialization library just to handle one particular type. That creates a sort of open ended maintainence issue. I would hope someone might suggest some elegant edition to the interface that would be guarenteed to resolve
The code that needs to be added inside the serialization library is not specific to shared_ptr at all, or any other type for that matter. It's just the code for a mechanism to store object references, keyed by pointers, that can then be used by 3rd party programmers to serialize eg smart pointers.
I really can't know that one solution is going to work for every "reasonable" smart pointer. and even so it would require adding every new type that came along into the serialization library. This would defeat the original purpose of factoring out this code into a "library"
As I was saying, it's not necessary to add anything type-specific to the serialization library, just the basic tools so that serialization of eg shared_ptr can be implemented externally, externally both to shared_ptr and to the serialization library. /Jarl
Hi, Robert Ramey wrote:
For what its worth, my thoughts along these lines are something like:
calss helper_base { virtual ~helper_base() = 0; }
class text_iarchive ... std::list<helpbase *> helper_list; text_iarchive(){ for(it = help_list.begin();it != helper_list.end();++it) delete *it; } };
This is a good idea, although it shouldn't be part of text_iarchive, but some base class of it instead, so that all archive types can use these 'helpers' as you call them. Some time ago, this was one of our feature requests for boost::serialization and we've implemented this in our version of the library. We currently call these 'helpers' 'archive state', for the lack of a better word. Basically, this mechanism allows applications to associate arbitrary objects with archives that can be accessed from serialization code. These objects are basically per-instance variables of an archive. This is not only useful for shared_ptr serialization, but for other things as well, such as integrating boost::serialization with a serialization system of a 3rd party library. Another thing we use this mechanism for is when de-serialized objects need to register themselves with some kind of global registry or other type of manager object. So we strongly encourage adding such a feature to boost::serialization. Best Regards, Martin TAB Austria Haiderstraße 40 4052 Ansfelden Austria Phone: +43 7229 78040-218 Fax: +43 7229 78040-209 E-mail: martin.ecker@tab.at http://www.tab.at
martin.ecker@tab.at wrote:
Hi,
Robert Ramey wrote:
For what its worth, my thoughts along these lines are something like:
calss helper_base { virtual ~helper_base() = 0; }
class text_iarchive ... std::list<helpbase *> helper_list; text_iarchive(){ for(it = help_list.begin();it != helper_list.end();++it) delete *it; } };
This is a good idea, although it shouldn't be part of text_iarchive, but some base class of it instead,
Of course, this was only an example/sketch for illustrative purposes.
so that all archive types can use these 'helpers' as you call them. Some time ago, this was one of our feature requests for boost::serialization and we've implemented this in our version of the library. We currently call these 'helpers' 'archive state', for the lack of a better word. Basically, this mechanism allows applications to associate arbitrary objects with archives that can be accessed from serialization code. These objects are basically per-instance variables of an archive. This is not only useful for shared_ptr serialization, but for other things as well, such as integrating boost::serialization with a serialization system of a 3rd party library. Another thing we use this mechanism for is when de-serialized objects need to register themselves with some kind of global registry or other type of manager object. So we strongly encourage adding such a feature to boost::serialization.
Your experience suggests this is on the right track. I would like to see this kicked around a while to see that it would really satisfy those who are unhappy about this aspect of serialization. Its also possible that someone can invent a more elegant, transparent, and typesafe way of doing I can live with it - question is can everyone else? Robert Ramey
How about posting the relevant code so we can contrast it with Peter's ? Robert Ramey martin.ecker@tab.at wrote:
Hi,
Robert Ramey wrote:
For what its worth, my thoughts along these lines are something like:
calss helper_base { virtual ~helper_base() = 0; }
class text_iarchive ... std::list<helpbase *> helper_list; text_iarchive(){ for(it = help_list.begin();it != helper_list.end();++it) delete *it; } };
This is a good idea, although it shouldn't be part of text_iarchive, but some base class of it instead, so that all archive types can use these 'helpers' as you call them. Some time ago, this was one of our feature requests for boost::serialization and we've implemented this in our version of the library. We currently call these 'helpers' 'archive state', for the lack of a better word.
Basically, this mechanism allows applications to associate arbitrary objects with archives that can be accessed from serialization code. These objects are basically per-instance variables of an archive. This is not only useful for shared_ptr serialization, but for other things as well, such as integrating boost::serialization with a serialization system of a 3rd party library. Another thing we use this mechanism for is when de-serialized objects need to register themselves with some kind of global registry or other type of manager object. So we strongly encourage adding such a feature to boost::serialization.
Best Regards, Martin
TAB Austria Haiderstraße 40 4052 Ansfelden Austria Phone: +43 7229 78040-218 Fax: +43 7229 78040-209 E-mail: martin.ecker@tab.at http://www.tab.at
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
martin.ecker@tab.at wrote:
Hi,
After fixing header include orders for the umpteenth time to ensure boost/serialization/shared_ptr.hpp is included before shared_ptr.hpp so it can do it's #define private public hack I wondered if this was any closer to an acceptable resolution? I know this is brought up periodcally but it really needs to be sorted out in someway
I fully agree. [...]
"Proper" serialization for shared_ptr (it should work for both boost:: and tr1:: versions) is on my to-do list, but I foresee heaps of non-Boost-related work in the coming days :-( So unfortunately I'll not be able to implement both lock-free and serialization in time for 1.33. Sorry.
"Proper" serialization for shared_ptr (it should work for both boost:: and tr1:: versions) is on my to-do list, but I foresee heaps of non-Boost-related work in the coming days :-(
No worries, more than understandable given my own current workload. thanks Martin -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.308 / Virus Database: 266.9.3 - Release Date: 5/04/2005
For now this is the best solution. Or maybe one might want to tweak the share_ptr header so to include a friend template boost::archive::serialize(.... Robert Ramey Martin Slater wrote:
Hi there,
I have just hacked my local boost source to make shared_ptr's private declarations public to avoid the issue but obviously this is far from ideal.
Thanks
Martin
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey wrote:
For now this is the best solution. Or maybe one might want to tweak the share_ptr header so to include a friend template boost::archive::serialize(....
I thought about this but decided not to glorify the hack;) For now it will just have to do. At least it is on the radar to be fixed at some point. cheers Martin -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.308 / Virus Database: 266.9.3 - Release Date: 5/04/2005
participants (7)
-
Christoph Ludwig -
David Abrahams -
Jarl Lindrud -
Martin Slater -
martin.ecker＠tab.at -
Peter Dimov -
Robert Ramey