[serialization][trunk] please bring us archive helpers back
Hello, Boost.Serialization 1.33-1.34 introduces the notion or "helpers" to the Archive interface by means of the member functions: void lookup_helper( const extended_type_info* const eti, shared_ptr<void>& sph); void insert_helper( const extended_type_info* const eti, shared_ptr<void>& sph); which allow the user to make an archive instance store arbitrary helper objects associated to any given type. This helper interface is used to implement serialization of boost::shared_ptr in a non-intrusive way, which was impossible for former versions of Boost.Serialization. This is a wonderful addition to the library because not only solves the old problem with serializating shared_ptr but also serves as a general utility for implementing complicated serialization schemes, notably those in which internal resources are serialized which cannot be directly manipulated from the class public interface. I have used archive helpers to good effect in implementing an otherwise unfeasible serialization algorithm for a library I'm writing, but when I moved from Boost 1.34 to the SVN trunk I discovered to my dismay that this (admittedly undocumented) feature has been removed in favor of an ad-hoc extension of the Archive interface (internally based on something resembling the former helper code) to specifically allow serialization of boost::shared_ptr. IMHO this is a step back because it singles out shared_ptr as a type receiving special attention (vg. what will happen with std::tr1::shared_ptr, will the interface be augmented again to cope with this?) and primarily because it deprives us developers from a very very powerful tool. Could the helper interface please please be brought back to Boost.Serialization? Thank you, Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
I put the archive helper in as an ad hoc solution to a particular problem. I was always unsatisfied with it as I didn't like the idea of decorating the main serialization library code with special cases. Its my belief that this is what makes libraries die of old age. However, at the time it seemed expedient so I included it for this one case (shared pointer) and left it undocumented on purpose. Eventually I found what to me is a much better solution and this is implemented in the next release. This is to use multiple inheritance to add the helper in as a mix-in. This is the way I plan to support this kind of thing in the future. It doesn't change the published API of the serialization library itself. However I will add a section in the documentation - if its not already there describing this technique. I believe that using mix-in to ad helpers will be equivalent to adding them in at runtime as they are now. I included the shared_ptr helper mix-in in the in the library in order to not change the current api and in recognition of shared_ptrs priviledged status as a boost component. I didn't expect anyone to notice this change, as no one had posted a problem which required helpers and it was undocumented. Robert Ramey Joaquín Mª López Muñoz wrote:
Hello,
Boost.Serialization 1.33-1.34 introduces the notion or "helpers" to the Archive interface by means of the member functions:
void lookup_helper( const extended_type_info* const eti, shared_ptr<void>& sph); void insert_helper( const extended_type_info* const eti, shared_ptr<void>& sph);
which allow the user to make an archive instance store arbitrary helper objects associated to any given type. This helper interface is used to implement
serialization of boost::shared_ptr in a non-intrusive way, which was impossible for former versions of Boost.Serialization. This is a wonderful addition
to the library because not only solves the old problem with serializating shared_ptr but also serves as a general utility for implementing complicated serialization schemes, notably those in which internal resources are serialized which cannot be directly manipulated from the class public interface.
I have used archive helpers to good effect in implementing an otherwise unfeasible serialization algorithm for a library I'm writing, but when I moved from Boost 1.34 to the SVN trunk I discovered to my dismay that this (admittedly undocumented) feature has been removed in favor of an ad-hoc
extension of the Archive interface (internally based on something resembling the former helper code) to specifically allow serialization of boost::shared_ptr. IMHO this is a step back because it singles out shared_ptr as a type receiving special attention (vg. what will happen with std::tr1::shared_ptr, will the interface be augmented again to cope with this?) and primarily because it deprives us developers from a very very powerful tool.
Could the helper interface please please be brought back to Boost.Serialization? Thank you,
Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey ha escrito:
I put the archive helper in as an ad hoc solution to a particular problem. I was always unsatisfied with it as I didn't like the idea of decorating the main serialization library code with special cases. Its my belief that this is what makes libraries die of old age.
As I see it, the helper interface does not litter the serialization code, it merely provides an extension API for users implementing non-trivial serialization schemes --note that the helper themselves are not part of B.S but belong in the different serializable types, much like the associated serialize() functions.
However, at the time it seemed expedient so I included it for this one case (shared pointer) and left it undocumented on purpose.
Eventually I found what to me is a much better solution and this is implemented in the next release. This is to use multiple inheritance to add the helper in as a mix-in. This is the way I plan to support this kind of thing in the future. It doesn't change the published API of the serialization library itself. However I will add a section in the documentation - if its not already there describing this technique.
I believe that using mix-in to ad helpers will be equivalent to adding them in at runtime as they are now.
I included the shared_ptr helper mix-in in the in the library in order to not change the current api and in recognition of shared_ptrs priviledged status as a boost component.
I didn't expect anyone to notice this change, as no one had posted a problem which required helpers and it was undocumented.
With all due respect (and admiration for your great lib) I think the mix-in idea has many flaws as compared with the former helper interface. Let me state several points supporting my thesis and addressing some of the claims you make above; I'd be grateful if you could discuss them with me: 1. You state that the mix-in approach "doesn't change the published API of B.S". I don't think so: if I decide to implement an Archive type I must explicitly handle shared_ptr (or decide not to do it, I guess this is what your naked_* archive versions are for), so definitely this shows through the API I'm expected to implement. 2. Having an ad-hoc extension for one particular type seems to me a recipe for disaster. Consider the following: some environments are already providing std::tr1::shared_ptr, which of course does not have built-in serialization capabilities. How am I supposed to serialize this? Are you going to add another mix-in to your Archive classes to handle this as well? 3. You say that adding mix-ins is equivalent to adding helpers, but I think the approaches are radically different. Addition of a mix-in is a. made at compile time b. part of the implementation of the archive while addition of a helper is a. made at run-time b. part of the serialization code for a type Mix-in are closed for extension where helpers are open: If I need a helper for serializing some T and I'm using an Archive whitout special support for that T, all hope is lost. 4. IMHO the helper API in B.S 1.34 is a simple yet powerful capability, and does not look forced or clumsy at all. It simply provides a way to add state info to an archive object, which is in a sense a state machine transtitioning its way during the serialization process. I contend that helpers allow for implementing serialization for types which otherwise would be impossible to serialize or should rely on inefficient detours: a. boost::shared_ptr (unless adhoc provisions are made) b. std::tr1::shared_ptr c. STL containers iterators. d. The lib I'm writing, which is the original cause why I started this thread, provides a class called flyweight<T> where flyweights with the same value share their representation by holding internal pointers to the same T value. Serializating this is trivial with the aid of a saving helper that maps T*s to flyweight<T>s --without helpers, the best I can come up with involves T duplication in the serialization stream and inefficient creation of flyweights on saving time. Of course I'm aware helpers were undocumented and I was basically using them at my own risk, so I'm not complaining about my code being broken by the switch from B.S 1.34 to trunk, I just want to convince you that helpers are indeed a useful general-purpose capability that should be made public, and that the proposed mix-in alternative is flawed. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
Joaquín Mª López Muñoz wrote:
As I see it, the helper interface does not litter the serialization code, it merely provides an extension API for users implementing non-trivial serialization schemes --note that the helper themselves are not part of B.S but belong in the different serializable types, much like the associated serialize() functions.
I didn't really see this. In large part due to the fact that no one over the years as expressed a need for it. The only reason I had to add it was to accomodate design issues with smart_ptr which makes it impossible to serialize without special help. So I always considered it a pathalogical case.
With all due respect (and admiration for your great lib) I think the mix-in idea has many flaws as compared with the former helper interface. Let me state several points supporting my thesis and addressing some of the claims you make above; I'd be grateful if you could discuss them with me:
First of all, let me say its always a pleasure to disagree with you.
1. You state that the mix-in approach "doesn't change the published API of B.S". I don't think so: if I decide to implement an Archive type I must explicitly handle shared_ptr (or decide not to do it, I guess this is what your naked_* archive versions are for), so definitely this shows through the API I'm expected to implement.
For that one case. I understand the situation, but as I said I felt that was an (almost) unique situation.
2. Having an ad-hoc extension for one particular type seems to me a recipe for disaster. Consider the following: some environments are already providing std::tr1::shared_ptr, which of course does not have built-in serialization capabilities. How am I supposed to serialize this? Are you going to add another mix-in to your Archive classes to handle this as well?
I haven't made any plans. Even the runtime helper, I would still have to make special code for a new shared_ptr - so its just a question of where the code ends up.
3. You say that adding mix-ins is equivalent to adding helpers, but I think the approaches are radically different. Addition of a mix-in is a. made at compile time b. part of the implementation of the archive while addition of a helper is a. made at run-time b. part of the serialization code for a type Mix-in are closed for extension where helpers are open: If I need a helper for serializing some T and I'm using an Archive whitout special support for that T, all hope is lost.
4. IMHO the helper API in B.S 1.34 is a simple yet powerful capability, and does not look forced or clumsy at all. It simply provides a way to add state info to an archive object, which is in a sense a state machine transtitioning its way during the serialization process. I contend that helpers allow for implementing serialization for types which otherwise would be impossible to serialize or should rely on inefficient detours: a. boost::shared_ptr (unless adhoc provisions are made) b. std::tr1::shared_ptr c. STL containers iterators. d. The lib I'm writing, which is the original cause why I started this thread, provides a class called flyweight<T> where flyweights with the same value share their representation by holding internal pointers to the same T value. Serializating this is trivial with the aid of a saving helper that maps T*s to flyweight<T>s --without helpers, the best I can come up with involves T duplication in the serialization stream and inefficient creation of flyweights on saving time.
As, I've said, based on experience so far, I've been of the idea that the only case where one would ever need this is shared_ptr. So we can think about this. Lately I've become more focused on improving performance of the serialization library and this provided part of the motivation for eliminating helpers for those who don't need them. Another very significant issue was that making the helper concept official would mean that I would have to document it which would lead to people asking questions about it which would lead to people spending a lot of time telling me how it could/should be enhanced, and me responding, etc, etc, etc. So I was and still am interested in shrinking down the serialization library by factoring that which is orthogonal to serialization. Perhaps the best resolution is to just make a mix-in class equivalent to the previous helper functionality. It would seem to me that this would give you every thing we both need. Robert Ramey
Robert Ramey ha escrito:
Joaquín Mª López Muñoz wrote:
As I see it, the helper interface does not litter the serialization code, it merely provides an extension API for users implementing non-trivial serialization schemes --note that the helper themselves are not part of B.S but belong in the different serializable types, much like the associated serialize() functions.
I didn't really see this. In large part due to the fact that no one over the years as expressed a need for it. The only reason I had to add it was to accomodate design issues with smart_ptr which makes it impossible to serialize without special help. So I always considered it a pathalogical case.
I think helpers (or some other extension mechanism) are required to implement non-intrusive serialization of classes with shared_ptr semantics --to put it more formally, copyable types T internally holding a pointer to some shared resource of type R where T is not constructible from a R* (or this construction does not have sharing semantics). This includes shared_ptr, the class I'm working on and I'm sure there are many other cases. It's not so usual, but it's bot pathological. [...]
As, I've said, based on experience so far, I've been of the idea that the only case where one would ever need this is shared_ptr.
So we can think about this.
Thank you for considering my petition. I'll try to show you that there's genuine value in having helpers (or some equivalent stuff) back besides solving my particular problem.
Lately I've become more focused on improving performance of the serialization library and this provided part of the motivation for eliminating helpers for those who don't need them.
Yep, I guess a mix-in helper for shared_ptr is faster than helpers because the latter incur the extended_type_info lookup for the helper, right? Anyway, you can have the helper interface for extensibility and the mix-in helper for shared_ptr and some other native types you'd wish to have built-in support for.
Another very significant issue was that making the helper concept official would mean that I would have to document it which would lead to people asking questions about it which would lead to people spending a lot of time telling me how it could/should be enhanced, and me responding, etc, etc, etc. So I was and still am interested in shrinking down the serialization library by factoring that which is orthogonal to serialization.
You can see it this way: if you provide the helper interface, next time someone (like me) comes whining at you about serialization of T not being possible with B.S standard API (as is the case for T=shared_ptr) you can answer him back "use the helper API" instead of burdening yourself with the task of providing native support for the type (as was the case for T=shared_ptr). Less work in the long run :)
Perhaps the best resolution is to just make a mix-in class equivalent to the previous helper functionality. It would seem to me that this would give you every thing we both need.
I'm not sure I'm getting this: would you mind elaborating this design? Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
Joaquín Mª López Muñoz wrote:
Robert Ramey ha escrito:
Perhaps the best resolution is to just make a mix-in class equivalent to the previous helper functionality. It would seem to me that this would give you every thing we both need.
I'm not sure I'm getting this: would you mind elaborating this design?
Hmmm - just to avoid stressing my brain by going into detail, let me just sketch it out. Under the 1.34 helper system, a few calls were added to the class hierarchy at high level. This basically added a set of pointers to helper classes which were handled by smart_ptr so they would automatically be destroyed when the archive was destroyed. If you want this functionality in he new system a) create a "helper manager" class b) move the helper manager code from the 1.34 release into this new class. c) mixin you new "helper manager" along with "naked_text_iarchive" and - voila - you have the old functionality as it was before. I'm sure this wouldn't be hard to do. Actually this is good enough idea that I might just do it in the library so that it would be the default but still leave the naked... archives available for those who don't use smart_ptr or other classes which need help. Food for thought. Robert Ramey
----- Mensaje original ----- De: Robert Ramey <ramey@rrsd.com> Fecha: Viernes, Septiembre 7, 2007 5:42 pm Asunto: Re: [boost][serialization][trunk]please bring us archive helpers back Para: boost@lists.boost.org
Perhaps the best resolution is to just make a mix-in class equivalent to the previous helper functionality. It would seem to me that this would give you every thing we both need. [...] Under the 1.34 helper system, a few calls were added to the class hierarchy at high level. This basically added a set of pointers to helper classes which were handled by smart_ptr so they would automatically be destroyed when the archive was destroyed.
If you want this functionality in he new system a) create a "helper manager" class b) move the helper manager code from the 1.34 release into this new class. c) mixin you new "helper manager" along with "naked_text_iarchive" and - voila - you have the old functionality as it was before.
I'm sure this wouldn't be hard to do.
This can be done, but makes the archive types to be used dependent on the types to be serialized!! This is a wrong approach, IMHO. Consider the following: I have a type foo for which I need a serialization helper, let's call it foo_ser_helper, which I embed into B.S by means of a foo_ser_hlp_manager as you suggest: struct foo_text_iarchive: naked_text_iarchive, foo_ser_hlp_manager {...}; Now, the problem is that user code willing to serialize my foo can't use a regular text_iarchive but have to resort to foo_text_iarchive instead. This is annoying enough because I force the user to change her code in a rather surprising way (considering she need not change archive types for serializing "normal" types), but things can get worse: if now some other author provides a class bar under similar conditions: struct bar_text_iarchive: text_iarchive, bar_ser_hlp_manager {...}; what should someone do in order to serialize the following data type? struct udt { foo f; bar b; }; So, I think providing archive types specialized for serializing specific types basically breaks down the architectural design of B.S, which is based on the separation of type-agnostic archives and archive-agnostic serialization code: this is too good to depart from. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
"JOAQUIN LOPEZ MU?Z" wrote:
So, I think providing archive types specialized for serializing specific types basically breaks down the architectural design of B.S, which is based on the separation of type-agnostic archives and archive-agnostic serialization code: this is too good to depart from.
I totally understand and am in total agreement. I made an exception to this general rule for shared_ptr which I felt I had to do at the time. But I think you missed the main point of my last message. The 1.33 system has an api for attaching helpers to to an archive class. This was only used (as far as I knew) and only needed to address the special problems associated with the serialization of shared_ptr. This introduced code into the library for handling one special case - which we both agree is a bad idea. Now you're telling me that you need it for other types - well that was news to me. So I'm thinking that a better solution might be to create a class "helper manager" which contains the functionality of the 1.33 and mixing in THIS class at the bottom of the hierarchy. So things would give you back the functionality of 1.33 while still removing inappropriate API decoration of the base classes. Robert Ramey
Joaquín M López Muñoz Telefónica, Investigación y Desarrollo _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
----- Mensaje original ----- De: Robert Ramey <ramey@rrsd.com> Fecha: Sábado, Septiembre 8, 2007 5:32 pm Asunto: Re: [boost] [serialization][trunk]please bring us archive helpers back Para: boost@lists.boost.org [...]
I totally understand and am in total agreement. I made an exception to this general rule for shared_ptr which I felt I had to do at the time.
But I think you missed the main point of my last message.
Yep, I guess so, excuse my lack of understanding. Please bear with me.
The 1.33 system has an api for attaching helpers to to an archive class. This was only used (as far as I knew) and only needed to address the special problems associated with the serialization of shared_ptr. This introduced code into the library for handling one special case - which we both agree is a bad idea. Now you're telling me that you need it for other types - well that was news to me.
So I'm thinking that a better solution might be to create a class "helper manager" which contains the functionality of the 1.33 and mixing in THIS class at the bottom of the hierarchy. So things would give you back the functionality of 1.33 while still removing inappropriate API decoration of the base classes.
So, if I'm getting you right now, what you propose is to have a naked_text_archive without helper support nor special shared_ptr support and text_archive with both helper and shared_ptr stuff? Is the goal of this move to have the helper API back in place without making it part of the Archive concept? Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
"JOAQUIN LOPEZ MU?Z" wrote:
So, if I'm getting you right now, what you propose is to have a naked_text_archive without helper support nor special shared_ptr support and text_archive with both helper and shared_ptr stuff?
naked archive wouldn't have support for helpers of any kind. This is as it is now. particular archives - text, binary, and xml would include helper support implemented as a mix-in which would re-implement the runtime helper support like 1.33
Is the goal of this move to have the helper API back in place without making it part of the Archive concept?
Yes ,that would be the object of this excercise. Note that I did things the way I did as I though the issues related to serialization of shared_ptr were very unique. I would be curious to know if anyone else had problems with this. Robert Ramey
Joaquín M López Muñoz Telefónica, Investigación y Desarrollo _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey ha escrito:
naked archive wouldn't have support for helpers of any kind. This is as it is now. particular archives - text, binary, and xml would include helper support implemented as a mix-in which would re-implement the runtime helper support like 1.33
Is the goal of this move to have the helper API back in place without making it part of the Archive concept?
Yes ,that would be the object of this excercise.
Note that I did things the way I did as I though the issues related to serialization of shared_ptr were very unique. I would be curious to know if anyone else had problems with this.
OK, now your proposal solves my particular problem, thank you for that. I'll try to present evidence that the helper API is useful enough to be promoted to the Archive concept. Give me some time to try to come back to you with compelling examples of use of this facility. Best, Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
participants (3)
-
"JOAQUIN LOPEZ MU?Z" -
Joaquín Mª López Muñoz
-
Robert Ramey