[serialization] Proposal for an extension API to the Archive concept
From the point of view of the implementer of an Archive, satisfying this new requirement is as simple as having an internal member of type shared_ptr<void> and granting access to it through member functions as described above. That's all the extra burden put on the Archive concept. So far,
Reviewing the discussion in the thread entitled "[serialization][trunk] please bring us archive helpers back" at http://lists.boost.org/Archives/boost/2007/09/127065.php , I've been thinking about the design issues raised there and I'd like to make a proposal for a seamless extension API that I think can solve the practical concerns originating the discussion while addressing Robert's policy of keeping the Archive concept simple. The last proposal Robert made regarding the helper API and special support for shared_ptr can be summarized in the following points: 1. The Archive concept does not include any extra requirement concerning special support for shared_ptr and/or a helper API. 2. There are naked_* archive implementations which conform to the bare requirements of the Archive concept. 3. The traditional archive implementations (without the "naked_" prefix) inherit from naked_ as well as from a class shared_ptr_helper and a class named, say, helper_api so that special support for shared_ptr and a helper API are provided. This approach has pros and cons: pros: the Archive concept is not polluted by impure considerations such as those accomodating shared_ptr and helpers, thus helping keep the conceptual design and the work of archive implementers simple. cons: A naked archive implementation, while conforming with the concept requirements, will fail to handle shared_ptr and any type relying on the helper API. My point of view is that this solution is just a half-baked one: having a conformant archive which fails to handle shared_ptrs would come as a surprising (and annoying) fact to users of B.S. So, how to make B.S provide shared_ptr and helper support whitout polluting the archive concept with too specific provisions? My proposal is to add a single "hook" to the Archive concept which is as agnostic as possible and can be leveraged in the future for unforeseen necessities beyond shared_ptr and helper support. Add the following requirement to the Archive concept (both Saving and Loading): a.get_extension_obj(p); a.set_extension_obj(p); Returns and sets, respectively, an internal object of type shared_ptr<void>. This object is default initialized upon archive construction and is destroyed along with destruction of the archive. These expressions are reserved for the internal usage of Boost.Serialization and thus must not be called by user code. the concept is not polluted with considerations on shared_ptrs, helpers or anything. How can Boost.Serialization leverage this extension API to *universally* provide helper and shared_ptr support for any Archive type? The following is an skectch of the procedure: B.S can define an internal type resembling this: class extension: public shared_ptr_helper, public helper_api {}; where shared_ptr_helper is more or less like the currently existing class and helper_api will package the helper API functions formerly present in Boost 1.34 (lookup_helper, insert_helper, etc.) Now, B.S can publicly provide the following helper API to user code: namespace boost{ namespace serialization{ template<typename Archive> void lookup_helper( Archive& ar, const extended_type_info* eti, shared_ptr<void> & sph); // rest of the helper API ... } } which is implemented along the following lines: template<typename Archive> void lookup_helper( Archive& ar, const extended_type_info* eti, shared_ptr<void> & sph) { // get the extension object or create a new one if this // is the first time shared_ptr<void> ext_obj=ar.get_extension_obj(); if(!ext_obj){ ext_obj.reset(new extension); ar.set_extension_obj(ext_obj); } extension& ext=*ext_obj; // forward to the extension object ext.lookup_helper(eti,sph); } and similarly for the other helper API and shared_ptr support functions. What do we gain with this? 1. The Archive concept is kept almost intact, and the required extension is both trivial to implement and totally agnostic. 2. We can provide support for shared_ptr and helpers *universally*, without putting the burden of its implementation to the writers of Archive classes. This extra functionality will be called using a notation boost::serialization::foo(ar,...); instead of ar.foo(...); but this is the only drawback (if it can be considered a drawback.) 3. Future functionality can be added by Boost.Serialization to Archives by adjoining it to the internal extension class without the Archive concept or its implementation needing any revision at all. How do you like it? Your feedback is greatly appreciated. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
"JOAQUIN LOPEZ MU?Z" wrote:
cons: A naked archive implementation, while conforming with the concept requirements, will fail to handle shared_ptr and any type relying on the helper API.
And will not force everyone who doesn't happen to be using shared_ptr to pay a penalty. This is pro for me.
My point of view is that this solution is just a half-baked one: having a conformant archive which fails to handle shared_ptrs would come as a surprising (and annoying) fact to users of B.S.
That's why I added the shared_ptr_helper to create the "official" boost archive compatible with the current (1.34) one. With the curent situation, the default is to included support for shared pointer whether its used or not. Users who don't want this can use naked_archive.
So, how to make B.S provide shared_ptr and helper support whitout polluting the archive concept with too specific provisions? My proposal is to add a single "hook" to the Archive concept which is as agnostic as possible and can be leveraged in the future for unforeseen necessities beyond shared_ptr and helper support. Add the following requirement to the Archive concept (both Saving and Loading):
a.get_extension_obj(p); a.set_extension_obj(p);
Returns and sets, respectively, an internal object of type shared_ptr<void>. This object is default initialized upon archive construction and is destroyed along with destruction of the archive. These expressions are reserved for the internal usage of Boost.Serialization and thus must not be called by user code.
From the point of view of the implementer of an Archive, satisfying this new requirement is as simple as having an internal member of type shared_ptr<void> and granting access to it through member functions as described above. That's all the extra burden put on the Archive concept. So far, the concept is not polluted with considerations on shared_ptrs, helpers or anything.
This sounds to me exactly the 1.34 version. Its just that the "hook" is undocumented as I considered it a hack required to overcome the problems associated with just one type - shared_ptr.
How can Boost.Serialization leverage this extension API to *universally* provide helper and shared_ptr support for any Archive type? The following is an skectch of the procedure: B.S can define an internal type resembling this:
class extension: public shared_ptr_helper, public helper_api {};
where shared_ptr_helper is more or less like the currently existing class and helper_api will package the helper API functions formerly present in Boost 1.34 (lookup_helper, insert_helper, etc.) Now, B.S can publicly provide the following helper API to user code:
namespace boost{ namespace serialization{
template<typename Archive> void lookup_helper( Archive& ar, const extended_type_info* eti, shared_ptr<void> & sph); // rest of the helper API ... } }
which is implemented along the following lines:
template<typename Archive> void lookup_helper( Archive& ar, const extended_type_info* eti, shared_ptr<void> & sph) { // get the extension object or create a new one if this // is the first time
shared_ptr<void> ext_obj=ar.get_extension_obj(); if(!ext_obj){ ext_obj.reset(new extension); ar.set_extension_obj(ext_obj); } extension& ext=*ext_obj;
// forward to the extension object ext.lookup_helper(eti,sph); }
and similarly for the other helper API and shared_ptr support functions.
What do we gain with this?
1. The Archive concept is kept almost intact, and the required extension is both trivial to implement and totally agnostic. 2. We can provide support for shared_ptr and helpers *universally*, without putting the burden of its implementation to the writers of Archive classes. This extra functionality will be called using a notation
boost::serialization::foo(ar,...);
instead of
ar.foo(...);
but this is the only drawback (if it can be considered a drawback.) 3. Future functionality can be added by Boost.Serialization to Archives by adjoining it to the internal extension class without the Archive concept or its implementation needing any revision at all.
How do you like it? Your feedback is greatly appreciated.
I proposed making this much simpler. make a helper api which is similar to that which 1.34 has. The only difference is that it is not implemented in the base class but rather as a mix in - similar to the way shared_ptr_helper is now. So we would have naked_archive archive = naked_archive + helper api and the old shared_ptr serialization would be put back in. The implemenation would be the same, its just taht the calles would be implemented in the mix-in rather than in the base. and of course the helper api would be available for any other schemes - or one could make his own wacho_archive = naked_archive + helper_api + wacko_api So we would have what we had before but without unnecessary decoration required for special cases. Robert Ramey
Joaquín M López Muñoz Telefónica, Investigación y Desarrollo _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Robert Ramey ha escrito:
"JOAQUIN LOPEZ MU?Z" wrote:
cons: A naked archive implementation, while conforming with the concept requirements, will fail to handle shared_ptr and any type relying on the helper API.
And will not force everyone who doesn't happen to be using shared_ptr to pay a penalty. This is pro for me.
With the {get|set}_extension_obj proposal, those who are not using shared_ptr do not pay a penalty -- the extension object is never created.
So, how to make B.S provide shared_ptr and helper support whitout polluting the archive concept with too specific provisions? My proposal is to add a single "hook" to the Archive concept which is as agnostic as possible and can be leveraged in the future for unforeseen necessities beyond shared_ptr and helper support. [...] This sounds to me exactly the 1.34 version. Its just that the "hook" is undocumented as I considered it a hack required to overcome the problems associated with just one type - shared_ptr.
There are important differences with respect to Boost 1.34. Hooking is done at run-time, whereas the Boost 1.34 solution was compile-time. Extension is created lazily, which means you don't pay for it if you're not using it. The extension class can grow in the future without Archive classes needing to be redefined or recompiled.
I proposed making this much simpler.
make a helper api which is similar to that which 1.34 has. The only difference is that it is not implemented in the base class but rather as a mix in - similar to the way shared_ptr_helper is now.
So we would have
naked_archive archive = naked_archive + helper api
and the old shared_ptr serialization would be put back in. The implemenation would be the same, its just taht the calles would be implemented in the mix-in rather than in the base.
Well, I think our respective positions are clear, and I don't want to beat this to death, but please answer the following: under your proposed scheme, does the type shared_ptr conform to the concept Serializable? Note that the only reasonably acceptable answers are "yes" and "no" :) Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
Joaquín Mª López Muñoz wrote:
This sounds to me exactly the 1.34 version. Its just that the "hook" is undocumented as I considered it a hack required to overcome the problems associated with just one type - shared_ptr.
There are important differences with respect to Boost 1.34. Hooking is done at run-time, whereas the Boost 1.34 solution was compile-time. Extension is created lazily, which means you don't pay for it if you're not using it. The extension class can grow in the future without Archive classes needing to be redefined or recompiled.
That's the way it is in 1.34
Well, I think our respective positions are clear,
Not to me.
and I don't want to beat this to death, but please answer the following: under your proposed scheme, does the type shared_ptr conform to the concept Serializable? Note that the only reasonably acceptable answers are "yes" and "no" :)
As I've said, I don't see ANY functional difference between what you propose and what's already implemented in 1.34. If I remember correctly, you're concerns were raised upon seeing the changes in the trunk - (aka 1.35). I'm proposing to go back to 1.34 (with a slight change in implementation - no change in interface). I can't see how that is not exactly what you wanted. Robert Ramey
Robert Ramey ha escrito:
Joaquín Mª López Muñoz wrote:
This sounds to me exactly the 1.34 version. Its just that the "hook" is undocumented as I considered it a hack required to overcome the problems associated with just one type - shared_ptr.
There are important differences with respect to Boost 1.34. Hooking is done at run-time, whereas the Boost 1.34 solution was compile-time. Extension is created lazily, which means you don't pay for it if you're not using it. The extension class can grow in the future without Archive classes needing to be redefined or recompiled.
That's the way it is in 1.34
Well, I think that the differences between the hook proposal and what was provided by Boost 1.34 are patent to you, so I take your statement as meaning that you don't deem these differences important rather than not being any difference at all. Please correct me if I'm wrong.
Well, I think our respective positions are clear,
Not to me.
OK, I can try to explain myself better :) If you don't grow tired of this discussion I'll be happy to elaborate my point as much as needed so that at least we are both fully knowledgeable of each others' proposals. Then the final call will be yours, of course.
and I don't want to beat this to death, but please answer the following: under your proposed scheme, does the type shared_ptr conform to the concept Serializable? Note that the only reasonably acceptable answers are "yes" and "no" :)
As I've said, I don't see ANY functional difference between what you propose and what's already implemented in 1.34. If I remember correctly, you're concerns were raised upon seeing the changes in the trunk - (aka 1.35). I'm proposing to go back to 1.34 (with a slight change in implementation - no change in interface). I can't see how that is not exactly what you wanted.
What you propose indeed reverts to the situation in Boost 1.34 and for that I thank you. It solves my particular needs, but what I'm proposing is IMHO an improvement on that. My prior question "does shared_ptr conform to the concept Serializable" is not a rhetoric one, and I ask you to consider it closely. Under your proposed scheme, you'll have archive implementations (naked_*) which 1. conform to the Archive concept AND 2. are not able to deal with shared_ptrs. So seems like shared_ptr is serializable but not always. The strict conclusion, actually, is that shared_ptr is not serializable, and that's what I'm trying to improve upon. My proposal seeks to provide shared_ptr support *universally* so that a user is guaranteed that she'll be able to serialize her shared_ptrs with any archive provided the archive models the Archive concept --period, no "special" archives required or anything. To sum it up, what I propose is to go from situation 1 (your proposal) where: 1. The Archive concept does not guarantee serializability of shared_ptrs and types relying on the helper API. 2. There are archive implementations which support an extension of the Archive concept by which shared_ptrs and helper API are supported. to situation 2 where: 1. The Archive concept is sufficient to guarantee that shared_ptrs and types relying on the helper API are serializable. Did I get my point through so far? I'm not trying to convince you now, only to make sure I have explained my motivations clearly enough. When that's the case I can go on and provide further arguments in favor of my proposal. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
To sum it up, what I propose is to go from situation 1 (your proposal) where:
1. The Archive concept does not guarantee serializability of shared_ptrs and types relying on the helper API. 2. There are archive implementations which support an extension of the Archive concept by which shared_ptrs and helper API are supported.
to situation 2 where:
1. The Archive concept is sufficient to guarantee that shared_ptrs and types relying on the helper API are serializable.
Did I get my point through so far? I'm not trying to convince you now,
Suppose we just reverted to 1.34 and documented the helper interace. How would you proposal be different than this? That's what I'm missing. Of course it would mean an expansion of the Archive Concept for just one data type. Another layer of concept for users to (mis)understand and ask about and whatever. And there will always be data types whose authors don't wish to expose enough functionality to be compatiable with boost serialization - so were does it end? And all this for just one such datatype so far? Oh well. Robert Ramey
Robert Ramey ha escrito:
To sum it up, what I propose is to go from situation 1 (your proposal) where:
1. The Archive concept does not guarantee serializability of shared_ptrs and types relying on the helper API. 2. There are archive implementations which support an extension of the Archive concept by which shared_ptrs and helper API are supported.
to situation 2 where:
1. The Archive concept is sufficient to guarantee that shared_ptrs and types relying on the helper API are serializable.
Did I get my point through so far? I'm not trying to convince you now,
Suppose we just reverted to 1.34 and documented the helper interace. How would you proposal be different than this? That's what I'm missing.
Well, it depends on whether you state that all conformant Archive types must be implemented by deriving from common_{i|o}archive or not. If derivation from common_{i|o}archive is a requisite for conformance to the Archive concept, then there would be little *conceptual* differences between that situation and my proposal. This is a question for you: In 1.34, is an archive type *required* to derive from common_{i|o}archive to comply with the Archive concept? I'll try to summarize here the different approaches, past, present and proposed, that we have discussed so far, along with their implications regarding serializability of shared_ptrs and other types relying on the helper API. I'd be grateful if you can review the table and confirm whether you agree with it (leaving aside which option you favor, I'm here mostly interested in a correct description of the options): A) Situation in 1.34 * Archive types are not required to derive from common_{i|o}archive. * common_{i|o}archive provides undocumentedly the helper API. * shared_ptr is not Serializable formally speaking. It happens to be serializable for common_{i|o}archive-derived archives only, through an undocumented interface. B) Situation in 1.34, bis * Archive types *are* required to derive from common_{i|o}archive. * common_{i|o}archive provides undocumentedly the helper API. * shared_ptr is Serializable. It happens to be serializable through an undocumented interface, so no other type can formally speaking use that API. C) Situation in 1.34 + documented helper API * Archive types are not required to derive from common_{i|o}archive. * common_{i|o}archive provides the documented helper API. * shared_ptr is not Serializable formally speaking. It happens to be serializable for common_{i|o}archive-derived archives only, as well as any other type relying on the public helper API. D) Situation in 1.34 + documented helper API, bis * Archive types *are* required to derive from common_{i|o}archive. * common_{i|o}archive provides the documented helper API. * shared_ptr and any type relying on the public helper API are Serializable. E) Situation in trunk * Conformant Archive types are allowed not to provide shared_ptr support (e.g. naked_* archives). * shared_ptr is not Serializable formally speaking. It happens to be serializable for some archives only, through an undocumented interface. * No helper API provided. F) Robert's proposal as of 2007/09/08 (http://tinyurl.com/2dfa3m ) * Conformant Archive types are allowed not to provide shared_ptr support or the helper API (e.g. naked_* archives). * Some archive types do provide shared_ptr and helper API, though this is not a requirement of the Archive concept. * shared_ptr and any type relying on the public helper API are not Serializable formally speaking. They happen to be serializable for some archive types only. G) Joaquín's proposal as of 2007/09/26 (http://tinyurl.com/2tuucq ) * Conformant Archive types are required to provide an extension API ({get|set}_extension_obj). This constitutes a revision of the Archive concept. * B.S provides shared_ptr support and the helper API. * shared_ptr and any type relying on the public helper API are Serializable. I've tried to describe the options as dispassionately as I could, so as to lay a common ground for further discussion. Do you see any error in the description? Are you satisfied with this rendering of the available options?
Of course it would mean an expansion of the Archive Concept for just one data type.
But an important one (and more are coming, e.g. std::tr1::shared_ptr). You are of course free to avoid expanding the Archive concept as in E) and F), but the logical implication of this is that shared_ptr is technically not serializable. Nothing wrong about that, if that's your declared intention, but users should know.
Another layer of concept for users to (mis)understand and ask about and whatever.
That's the job of a lib maintainer. I hope you assess every possible extension to your library on the basis of its technical merits. If you plainly state that B.S is closed for extension, well, this discussion is moot.
And there will always be data types whose authors don't wish to expose enough functionality to be compatiable with boost serialization - so were does it end? And all this for just one such datatype so far?
In my particular case, I can assure you that the need of a helper API is a genuine one: I cannot serialize my type efficiently without that API, no matter how much I want to expose the guts of my type for serialization purposes. Please believe me on that, this is not a case of me wanting to keep my type encapulated, "pure", or anything. It's sheer impossibility to do it otherwise. Of course, coming up with some more types that could put the helper API to good use would strengthen my position, that' for sure. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
I've tried to describe the options as dispassionately as I could, so as to lay a common ground for further discussion. Do you see any error in the description? Are you satisfied with this rendering of the available options?
A very good summary and explanation.
But an important one (and more are coming, e.g. std::tr1::shared_ptr). You are of course free to avoid expanding the Archive concept as in E) and F), but the logical implication of this is that shared_ptr is technically not serializable. Nothing wrong about that, if that's your declared intention, but users should know.
OK - now I see better your concern. I would say: shared_ptr as concieved and implemented by boost does not provide sufficient exposure to be support the concept of Serializable Type as defined by Boost Serialization: Given its importance, various hacks have been made to address it, but they are still basically hacks. They've resulted in code which requires coupling of the two classes/concepts. The current and proposed resolution of this situtation requires is at best inefficient. The original implemenation of serialization for boost smart_ptr, was more twice as fast but depended on the implementation. The current one is slower, but but depends only upon the published interface. To really resolve this situation in a definitive manner, either Serializable/Archive Concept or shared pointer public interface have to modified. In effect our hack for shared_ptr extends the Archive Concept for this one specific case. But at least its optional and extends the concept by composition so that the original Concept remains intact and the hack is only visible, confusing, illlogical and opaque in the context of that one type. And someone comes up with a serializable shared_ptr then this can be dropped without having left any collateral damage in the Archive Concept. shared_ptr is the only type which has come up in several years which has this problem. Some application specific types might have this issue, but in those cases, I would expect the ability to attach an application specific helper to the archive, thereby expanding the Archive Concept for just that application would be acceptable. Of course these observations would apply to TR1::shared pointer as well as other serialization systems.
Another layer of concept for users to (mis)understand and ask about and whatever.
That's the job of a lib maintainer.
LOL - well, that's me. And really I'm not only trying to keep the job from getting bigger, I'm trying to make it smaller.
I hope you assess every possible extension to your library on the basis of its technical merits. If you plainly state that B.S is closed for extension, well, this discussion is moot.
The power of any software (or mathematical/logical ) module resides in its logical coherence. Arbitraritly extending it in tangent directions requires that the a use of the module or concept consider a bunch of special cases every time he uses even for simple cases. Such types of extensions result in a net reduction in the utility of the library. The best way to extend this library (or any other) is by adding specific behavior/facilities on the current one. C++ supports this through inheritance. My latest attempt at handling the shared pointer issue reflects this point of view. Remember, this whole issue has come about because shared_ptr (unlike any other type so far) has been written in a manner that it is effectively closed for extension. Maybe you want to direct some observations in that direction. I've just come accross a very relevent instance of this. the 1.35 version of the binary In order to support optimization of a couple of collection types extra machinery was added to basic_binary_?archive (in 1.35) This was subject of a fairly acrimonious discussion last year. This was packaged along with some needed improvements in the handling of collection. This enhancement basically breaks the Archive Concept for these types. I actually overlooked this aspect at the time. But in the last month in the course of doing an exhaustive test on portable_binary_?archive I found that it was broken. This was due to the fact that it was derived from binary_archive and that binary_archive interface was extended at the base level to handle these special cases. Among other things, this made portable_binary_archive useless as an example of how to extend and archive through derivation. So now I have to drop the example from the documentation and conjure up a new one, explain why extension through deriviation shouldn't be done for binary archive and make a new example based on another archive class. A huge net loss. And I had to make a new portable binary archive which didn't derive from basic_binary_archive. I had to spend a whole day to discover the problem which only showed up on a couple of tests. All this from a simple optimization which "everyone" needs.
And there will always be data types whose authors don't wish to expose enough functionality to be compatiable with boost serialization - so were does it end? And all this for just one such datatype so far?
In my particular case, I can assure you that the need of a helper API is a genuine one: I cannot serialize my type efficiently without that API, no matter how much I want to expose the guts of my type for serialization purposes. Please believe me on that, this is not a case of me wanting to keep my type encapulated, "pure", or anything. It's sheer impossibility to do it otherwise.
Hmm - on one hand, I don't doubt your sincerity nor your assessment. On the other hand, I've yet to see such a type myself. The original serialization of shared_ptr was implemented by exposing more of the shared_ptr internals so in that one case it is possible. Of course whether that solution is desirable would be a matter for discussion.
Of course, coming up with some more types that could put the helper API to good use would strengthen my position, that' for sure.
So the current situation, is a) We have an Archive Concept, and Serializable Concept which are fairly coherent. The are currently only broken by basic_binary?archive. b) The classes in the library common_?archve, basic_?archive, are implementation features. It is not required to derive from any of these classes to implement the concepts. Of course its convenient to do so as they implement common aspects of the Archive Concept - but they are not required to. c) Some types - so far only shared_ptr and your ? - do not conform to the concept of a serializable type as they stand. My position is i) they are very infrequent. ii) the Archive Concept can be extended for these special cases through composition (inheritance) to provide ad hoc solutions. That's what 1.34 did for shared_ptr and I'm willing to continue doing that. The only think i want to do is to move the helper API out of the basic_?archive where it pollutes the Archive concept (which is why i didn't document it) and package it as a mix-in which is used with naked_?archive to produce the "shared_ptr" friendly archive classes. What do you see wrong with that? Robert Ramey
----- Mensaje original ----- De: Robert Ramey <ramey@rrsd.com> Fecha: Viernes, Septiembre 28, 2007 6:06 pm Asunto: Re: [boost] [serialization] Proposal for an extension API to the Archive concept Para: boost@lists.boost.org
I've tried to describe the options as dispassionately as I could, so as to lay a common ground for further discussion. Do you see any error in the description? Are you satisfied with this rendering of the available options?
A very good summary and explanation.
Great, I'm glad we've got the discussion grounds agreed upon.
But an important one (and more are coming, e.g. std::tr1::shared_ptr).> You are of course free to avoid expanding the Archive concept as in E) and F), but the logical implication of this is that shared_ptr is technically not serializable. Nothing wrong about that, if that's your declared intention, but users should know.
OK - now I see better your concern. I would say:
shared_ptr as concieved and implemented by boost does not provide sufficient exposure to be support the concept of Serializable Type as defined by Boost Serialization: [...] shared_ptr is the only type which has come up in several years which has this problem.
Some application specific types might have this issue, but in those cases, I would expect the ability to attach an application specific helper to the archive, thereby expanding the Archive Concept for just that application would be acceptable.
OK, I understand your position on the grounds that you deem shared_ptr (and other potential types on the same vein) as pathological cases. I will try in the lines below to convince you (or at least instill some drops of doubt in you) that this is not necessarily the case. [...]
The power of any software (or mathematical/logical ) module resides in its logical coherence. Arbitraritly extending it in tangent directions requires that the a use of the module or concept consider a bunch of special cases every time he uses even for simple cases. Such types of extensions result in a net reduction in the utility of the library.
Some remarks on this: I cannot but wholeheartedly agree with you that logical coherence must drive the extension of any firmly grounded design. But from my point of view, the helper API is a perfectly sound addition to B.S, because it fullfills a very general need in very general fashion: The helper API is just about keeping state information associated to the serialization process of objects of a given type. Think about it: as it's currently modeled by B.S, serialization is esentially a stateless process: when an object t of type T is about to be serialized, the assumption is that it won't rely on other T instances which have been previously serialized. Is this assumption reasonable? Well, I contend that in some, non-pathological situations, stateful serialization is needed. An almost insultingly obvious case: object tracking. If you give me the helper API and an archive without object tracking, I can implement object tracking myself without relying on the archive implementation or B.S facilities --and it's very easy to do, if you think about it. How's that for applicability of the helper API? Down below I'm explaining also the type of mine which has originated the discussion, for another example of use of the helper API. [...]
Remember, this whole issue has come about because shared_ptr (unlike any other type so far) has been written in a manner that it is effectively closed for extension. Maybe you want to direct some observations in that direction.
That's fair criticism. But consider that there are cases where the type to be serialized is closed beyond the user's control. And my example below is open yet not serializable without helper API.
I've just come accross a very relevent instance of this. the 1.35 version of the binary
In order to support optimization of a couple of collection types extra machinery was added to basic_binary_?archive (in 1.35) This was subject of a fairly acrimonious discussion last year.
Can you provide me with a link to that thread? Thanks! [...]
In my particular case, I can assure you that the need of a helper API is a genuine one: I cannot serialize my type efficiently without that API, no matter how much I want to expose the guts of my type for serialization purposes. Please believe me on that, this is not a case of me wanting to keep my type encapulated, "pure", or anything. It's sheer impossibility to do it otherwise.
Hmm - on one hand, I don't doubt your sincerity nor your assessment. On the other hand, I've yet to see such a type myself. The original serialization of shared_ptr was implemented by exposing more of the shared_ptr internals so in that one case it is possible. Of coursewhether that solution is desirable would be a matter for discussion.
I wanted to keep this discussion untied from my particular problem, but you're entitled to see the case and evaluate by yourself, so here it goes, the following is a simplified description of the type to keep things short. My type implements a sort of flyweight idiom, by which objects with the same value internally keep a pointer to the same representation, so as to avoid duplication of data and excessive memory consumption: flyweight<string> fw("hello"); flyweight<string> fw2("hello"); // fw and fw2 internally have pointers to the same string object. A crude approximation to the implementation of flyweight is: template<typename T, template <typename> Container=...> class flyweight { private: typedef Container<T> factory_type; // used to keep value objects static factory_type factory; // global value factory // A flyweight maintains an iterator to the associated value typedef typename factory_type::iterator handle_type; handle_type handle; public: flyweight(const T& t) // ctor #1 { // retrieve an iterator to an equivalent value or else // insert a new one if no equivalent value is found h=factory.insert(t).first; } flyweight(const flyweight& x) // ctor #2 { // point to the same value as x h=x.h; } ... }; Now, I want to serialize flyweight *efficiently*. The first naive approach is this one: template<class Archive,T,...> void save(Archive& ar,const flyweight<T,...> & fw,const unsigned int) { ar<<*(fw.h); // serialize the associated value } template<class Archive,T,...> void load(Archive& ar,flyweight<T,...> & fw,const unsigned int) { T t; ar>>t; fw=flyweight<T,...>(t); } but this is not efficient because, on loading time, duplicate values are created through ctor #1, which incurs a factory lookup, when I'd want to use ctor #2 (direct copy from a previously loaded equivalent flyweight object). So that's the problem. In the particular case where handle_type is a pointer, the thing can be done by B.S object tracking, but as the type is an unspecified iterator I cannot do that. To me, this is a clear example of serialization needing state info. With the helper API, serializing flyweight<T> is implemented so efficiently and beatifully that it almost hurts :) [...]
So the current situation, is
a) We have an Archive Concept, and Serializable Concept which are fairly coherent. The are currently only broken by basic_binary?archive. b) The classes in the library common_?archve, basic_?archive, are implementation features. It is not required to derive from any of theseclasses to implement the concepts. Of course its convenient to do so as they implement common aspects of the Archive Concept - but they are not required to.
c) Some types - so far only shared_ptr and your ? - do not conform to the concept of a serializable type as they stand. My position is
i) they are very infrequent. ii) the Archive Concept can be extended for these special cases throughcomposition (inheritance) to provide ad hoc solutions.
I hope I've been able to cast some doubt on your commitment to i) Also, I'd like to add that, from my experience as a lib maintainer, functionality usually predates usage: it is not until you provide some new stuff that people begin seeing application scenarios for it, not the other way around. Much more so if you're keeping an advanced lib as Boost libs are held to be, where contributors for new ideas are scarcer.
That's what 1.34 did for shared_ptr and I'm willing to continue doing that. The only think i want to do is to move the helper API out of the basic_?archive where it pollutes the Archive concept (which is why i didn't document it) and package it as a mix-in which is used with naked_?archive to producethe "shared_ptr" friendly archive classes.
What do you see wrong with that?
About this last point of yours, so according to your proposals what archive types would provide the helper API? You say basic_?archive won't provide it? Was it this way in 1.34? If not, won't you be breaking shared_ptr serialization code then? I look forward to your opinions about my position on the generality of the helper API and about the flyweight case. In general, I understand your position and I think it's a reasonable one, given the particular weights you assign to the factors involved --different to mine. When we come to this point the thing it's not then about hard facts but opinions, but I hope I'll be able to pile some more arguments to my tip of the balance Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
"JOAQUIN LOPEZ MU?Z" wrote:
I look forward to your opinions about my position on the generality of the helper API and about the flyweight case. In general, I understand your position and I think it's a reasonable one, given the particular weights you assign to the factors involved --different to mine. When we come to this point the thing it's not then about hard facts but opinions, but I hope I'll be able to pile some more arguments to my tip of the balance
I've given a cursory examination of you email and don't have time right now to give the answer it merits. However, I'm not really looking to convince you that you have to give up something. I expect to propose what you want factored in two pieces: a) an implemenation of a "pure" archive concept b) a mechanism for extending any class modeling the "pure" concept above to address situations such as you describe. Whether or not the "officially" supported boost archive classes (text_?archive, xml_?archive, etc. ) should include one or more such extensions would be a separate discussion. This is a question that I am less concerned about personally. Robert Ramey
----- Mensaje original ----- De: Robert Ramey <ramey@rrsd.com> Fecha: Sábado, Septiembre 29, 2007 0:34 am Asunto: Re: [boost] [serialization] Proposal for an extension API to the Archive concept Para: boost@lists.boost.org
"JOAQUIN LOPEZ MU?Z" wrote:
I look forward to your opinions about my position on the generality of the helper API and about the flyweight case. In general, I understand your position and I think it's a reasonable one, given the particular weights you assign to the factors involved --different to mine. When we come to this point the thing it's not then about hard facts but opinions, but I hope I'll be able to pile some more arguments to my tip of the balance
I've given a cursory examination of you email and don't have time right now to give the answer it merits.
Please come back then when you have the time. Your elaborate feedback is greatly appreciated, and we're not in a hurry to get this sorted out.
However, I'm not really looking to convince you that you have to give up something. I expect to propose what you want factored in two pieces: a) an implemenation of a "pure" archive concept b) a mechanism for extending any class modeling the "pure" concept above to address situations such as you describe. Whether or not the "officially" supported boost archive classes (text_?archive, xml_?archive, etc. ) should include one or more such extensions would be a separate discussion. This is a question that I am less concerned about personally.
Robert Ramey
Joaquín M López Muñoz Telefónica, Investigación y Desarrollo PS: It's a pity nobody else has jumped into the discussion :(
I think there is a significant issue with your proposed extension mechanism: an archive can only contain a single extension object at a time. Consequently, if two types require different extension objects, they cannot both be serialized using the same archive. In particular, if say flyweight requires some special extension object, and shared_ptr requires its own special extension object, it won't be possible to serialize both using the same archive. It seems, therefore, that in practice there would have to be a single standardized extension object, which is either in use or not in use for a particular archive. -- Jeremy Maitin-Shepard
I think there is a significant issue with your proposed extension mechanism: an archive can only contain a single extension object at a time. Consequently, if two types require different extension objects, they cannot both be serialized using the same archive. In particular, if say flyweight requires some special extension object, and shared_ptr requires its own special extension object, it won't be possible to serialize both using the same archive. It seems,
Hi Jeremy, ----- Mensaje original ----- De: Jeremy Maitin-Shepard <jbms@cmu.edu> Fecha: Lunes, Octubre 8, 2007 10:15 pm Asunto: Re: [boost] [serialization] Proposal for an extension API to theArchive concept Para: boost@lists.boost.org therefore,
that in practice there would have to be a single standardized extension object,which is either in use or not in use for a particular archive.
My proposal allows for any type to register and use its own helper object: note that the proposed extension interface: a.get_extension_obj(p); a.set_extension_obj(p); is *not* meant to be public; quoting from http://tinyurl.com/2tuucq : "These expressions are reserved for the internal usage of Boost.Serialization and thus must not be called by user code." It is then Boost.Serialization that adjoins its own extension object to an Archive and publicizes a helper API based upon this internal object: template<typename Archive> void insert_helper( Archive& ar, const extended_type_info* eti, shared_ptr<void> & sph) { // get the extension object or create a new one if this // is the first time shared_ptr<void> ext_obj=ar.get_extension_obj(); if(!ext_obj){ ext_obj.reset(new extension); ar.set_extension_obj(ext_obj); } extension& ext=*ext_obj; // forward to the extension object ext.insert_helper(eti,sph); } So, each distinct T can register its separate helper object more or less like this: serialization::extended_type_info* eti= serialization::type_info_implementation<helper<T> >:: type::get_instance(); shared_ptr<void> sph; insert_helper(ar,eti,sph); ... Did I make myself clearer now? Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
I am sorry. I know this is unrelated to archive but what ever happened to flyweight. -----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of Jeremy Maitin-Shepard Sent: Monday, October 08, 2007 4:14 PM To: boost@lists.boost.org Subject: Re: [boost] [serialization] Proposal for an extension API to theArchive concept I think there is a significant issue with your proposed extension mechanism: an archive can only contain a single extension object at a time. Consequently, if two types require different extension objects, they cannot both be serialized using the same archive. In particular, if say flyweight requires some special extension object, and shared_ptr requires its own special extension object, it won't be possible to serialize both using the same archive. It seems, therefore, that in practice there would have to be a single standardized extension object, which is either in use or not in use for a particular archive. -- Jeremy Maitin-Shepard _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
----- Mensaje original ----- De: Jarrad Waterloo <jwaterloo@dynamicquest.com> Fecha: Martes, Octubre 9, 2007 9:10 pm Asunto: Re: [boost] [serialization] Proposal for an extension API to the Archive concept Para: boost@lists.boost.org
I am sorry. I know this is unrelated to archive but what ever happened to flyweight.
Hello Jarrad, I've been growing the flyweight lib during these last months and it's now in a fairly complete shape, I hope to be able to upload a preview version and prompt for interest after Boost 1.35. If you're curious drop me a private mail and I'll send you the stuff. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
Sorry, I have been on vacation. What is your email as I am interested? -----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org] On Behalf Of "JOAQUIN LOPEZ MU?Z" Sent: Tuesday, October 09, 2007 6:03 PM To: boost@lists.boost.org Subject: Re: [boost] [serialization] Proposal for an extension API to the Archive concept ----- Mensaje original ----- De: Jarrad Waterloo <jwaterloo@dynamicquest.com> Fecha: Martes, Octubre 9, 2007 9:10 pm Asunto: Re: [boost] [serialization] Proposal for an extension API to the Archive concept Para: boost@lists.boost.org
I am sorry. I know this is unrelated to archive but what ever happened to flyweight.
Hello Jarrad, I've been growing the flyweight lib during these last months and it's now in a fairly complete shape, I hope to be able to upload a preview version and prompt for interest after Boost 1.35. If you're curious drop me a private mail and I'll send you the stuff. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
participants (5)
-
"JOAQUIN LOPEZ MU?Z" -
Jarrad Waterloo -
Jeremy Maitin-Shepard -
Joaquín Mª López Muñoz
-
Robert Ramey